In this first email, we have summarized the four key topics that prospective Snowflake customers should know to give them some foundation and confidence to get the project rolling.

Migration Goals and Strategy

We always start with a statement that there is no one-size-fits-all migration strategy. Each organization should treat itself as an individual and not merely copy practices from others.

Future clients should be aware, that not all the pathways will get them to the point where they will be able to harness all Snowflake benefits at once. The choice of the most suitable approach depends on what primary goals they would like to achieve. For example, if they are running out of storage and they have a tight deadline because they don’t want to invest in new costly hardware, then lift ‘n’ shift of their existing data warehouse as-is, would be a better option to start with than complete re-engineering.

However, if their main goal is to take advantage of cloud-native capabilities that Snowflake has to offer, redesign would be hard to avoid. For example, using an existing on-prem data integration tool for data transformation will not exploit the potential of the cloud computing power as it could.

Furthermore, since they are traversing to Snowflake, they would want to immediately start taking advantage of its unlimited computing power and unlimited storage. For this reason, ELT approach is way more beneficial than traditional ETL since it allows them to quickly load all their data into the Snowflake data lake and perform all the transformations there. This way, they can also exploit solutions provided by Snowflake community, Snowflake Snowpipe, etc.

Proof of Concept

Before they choose their future data platform, they should run a POC use case on their data. This will allow them to learn and grasp the value a new solution can bring. Tangible results will also build confidence among stakeholders and users. Nevertheless, they should make sure that they choose the use case carefully so that it will assure they that Snowflake platform can support their requirements and (probably, as seen so many times), excel their expectations. 

Migration Plan

The complexity of data migration projects varies, but it can take up to several years for a large-scale project to complete. Devoting enough time to prepare a good data migration plan that will minimize disruptions and mitigate the risks, is crucial. Here is just a short checklist of the most important things they should have in mind (we will go deeper into some of them in the following emails).

• Scope.
  It is not only the data that needs to be migrated. There are multiple sources and DW staging areas, ETL processes in a variety of forms, metadata, security schemas and authorization privileges, orchestration processes, BI tools, users, and other data consumers. And none of them should be forgotten.
• Roadmap.
  If possible, they should avoid a Big Bang approach. Instead of only one and late go-live, they can mitigate the risk by trying to break the scope into several independent parts that can be migrated and released in phases.
• Architecture.
  In the case of large projects, two-phased architecture is something that should be taken into consideration as it might bring better value because they will be able to reap cloud-based benefits sooner.
• Process.
  Following a selected approach, the migration process should be clearly defined with a detailed checklist and go-live strategy, supported by strong project management.
• Team.
  Different skill sets are required during the migration project. From specialists knowing all the technical details related to existing ETL workload and data sources to business users who know the data itself and will execute user acceptance testing as the final step before production deployment.

Key Challenges

Regardless of the project size, challenges are very similar, so it is important to prepare the client what can come along the way, so they don’t come back to you because things didn’t go as planned ?. 

• Unreal scope, timeline, and budget.
  Preconditions for preparing a well-made data migration plan and defining a realistic scope, timeline and budget are a detailed analysis and a full assessment of their existing system. Over-the-thumb estimates can create a completely distorted picture of the required effort. They must be aware of the consequences that may result from neglecting this step.
• Complexity.
  As mentioned previously, migration will touch everything from data sources and ETL procedures to BI applications and user permissions and everything in between. The more the existing system is customized and the longer it is in use, the more attention is expected to be needed.
• Lack of human resources or knowledge.
  Missing knowledge about any component that is part of the migration, legacy application developers who are not available or busy project team members that are occupied with other activities may all have a major effect on meeting the deadlines. Don’t underestimate the effort that will be required from their part.
• Poor data quality. If the selected approach includes data warehouse modernization, they should be prepared for a parallel project dedicated to data quality.
• Subject matter expertise.
  Data migration is not something that is done regularly within a company and so it is important that they are informed about modern data migration approaches when hiring external consultants.

An important note here is, that data migration is not only an IT project. It requires as much collaboration from the business side as it does from IT. Many migrations miss the desired outcome solely because the involvement of the business teams was underestimated. They know the data that is to be migrated better than anyone else.

Why is understanding the data crucial for project success? Because you need to know which answers the data can give you and how you’ll get them. Furthermore, almost any data migration is usually accompanying a larger modernization project. It is therefore not enough that data is just moved as-is and sooner or later, the structure will need to be reviewed in a way that it will best fit into the new system.

The bad news is that there are no shortcuts. Data migrations are the times when skeletons come out of the closet, as all the traces of history are discovered. The more the existing system is customized and the longer it is already in use, the more attention is expected to be needed. Some of the data might need cleansing, some might need restructuring, and some will simply become obsolete and should be discarded. Time shouldn’t be wasted on migrating everything. The one thing you don’t want in the new system, is the clutter. That is why it is very common that a data quality project runs in parallel. This is what takes up the most time and team effort and in order to be done well the data owners must dig in. They are also usually the ones who prepare the specifications, provide various definitions or explanations, and do user acceptance tests.

As you can see, numerous important activities can be accomplished only by in-house resources. But because migrations are not something that organizations do frequently, they usually involve external professionals. The reasons are many, from having limited internal resources at their disposal or lacking subject matter expertise to not knowing the target system well enough or not possessing the tools that would automate and accelerate the migration process.

And when they turn to someone for help, it’s crucial that besides technical expertise, the team on the other side understand the business and industry specifics as well. Only then can be analytical challenges solved in the most meaningful way.

If you’ll have questions regarding the Financial, Energy, or Telecom Industry, please don’t hesitate to contact me at klemen.logar@in516ht.com and I’ll connect you to a member of our team.