In this first email, we have summarized the four key topics that prospective Snowflake customers should know to give them some foundation and confidence to get the project rolling.
Migration Goals and Strategy
We always start with a statement that there is no one-size-fits-all migration strategy. Each organization should treat itself as an individual and not merely copy practices from others.
Future clients should be aware, that not all the pathways will get them to the point where they will be able to harness all Snowflake benefits at once. The choice of the most suitable approach depends on what primary goals they would like to achieve. For example, if they are running out of storage and they have a tight deadline because they don’t want to invest in new costly hardware, then lift ‘n’ shift of their existing data warehouse as-is, would be a better option to start with than complete re-engineering.
However, if their main goal is to take advantage of cloud-native capabilities that Snowflake has to offer, redesign would be hard to avoid. For example, using an existing on-prem data integration tool for data transformation will not exploit the potential of the cloud computing power as it could.
Furthermore, since they are traversing to Snowflake, they would want to immediately start taking advantage of its unlimited computing power and unlimited storage. For this reason, ELT approach is way more beneficial than traditional ETL since it allows them to quickly load all their data into the Snowflake data lake and perform all the transformations there. This way, they can also exploit solutions provided by Snowflake community, Snowflake Snowpipe, etc.
Proof of Concept
Before they choose their future data platform, they should run a POC use case on their data. This will allow them to learn and grasp the value a new solution can bring. Tangible results will also build confidence among stakeholders and users. Nevertheless, they should make sure that they choose the use case carefully so that it will assure they that Snowflake platform can support their requirements and (probably, as seen so many times), excel their expectations.
The complexity of data migration projects varies, but it can take up to several years for a large-scale project to complete. Devoting enough time to prepare a good data migration plan that will minimize disruptions and mitigate the risks, is crucial. Here is just a short checklist of the most important things they should have in mind (we will go deeper into some of them in the following emails).
Regardless of the project size, challenges are very similar, so it is important to prepare the client what can come along the way, so they don’t come back to you because things didn’t go as planned ?.
An important note here is, that data migration is not only an IT project. It requires as much collaboration from the business side as it does from IT. Many migrations miss the desired outcome solely because the involvement of the business teams was underestimated. They know the data that is to be migrated better than anyone else.
Why is understanding the data crucial for project success? Because you need to know which answers the data can give you and how you’ll get them. Furthermore, almost any data migration is usually accompanying a larger modernization project. It is therefore not enough that data is just moved as-is and sooner or later, the structure will need to be reviewed in a way that it will best fit into the new system.
The bad news is that there are no shortcuts. Data migrations are the times when skeletons come out of the closet, as all the traces of history are discovered. The more the existing system is customized and the longer it is already in use, the more attention is expected to be needed. Some of the data might need cleansing, some might need restructuring, and some will simply become obsolete and should be discarded. Time shouldn’t be wasted on migrating everything. The one thing you don’t want in the new system, is the clutter. That is why it is very common that a data quality project runs in parallel. This is what takes up the most time and team effort and in order to be done well the data owners must dig in. They are also usually the ones who prepare the specifications, provide various definitions or explanations, and do user acceptance tests.
As you can see, numerous important activities can be accomplished only by in-house resources. But because migrations are not something that organizations do frequently, they usually involve external professionals. The reasons are many, from having limited internal resources at their disposal or lacking subject matter expertise to not knowing the target system well enough or not possessing the tools that would automate and accelerate the migration process.
And when they turn to someone for help, it’s crucial that besides technical expertise, the team on the other side understand the business and industry specifics as well. Only then can be analytical challenges solved in the most meaningful way.
If you’ll have questions regarding the Financial, Energy, or Telecom Industry, please don’t hesitate to contact me at email@example.com and I’ll connect you to a member of our team.