Back

Data Migration to Snowflake: A Well-made Migration Plan

19.01.2021

Data warehouse migration is usually not a project one would look forward to. It is often even unwanted, but it must be done as a precondition for completing a larger innovation or modernization project. However, the quality of migration is closely linked to the success of the latter.

Large-scale migration projects can take up to several years to complete. Devoting enough time to prepare a good data migration plan that will minimize disruptions and mitigate the risks, is key to success. The migration plan must be aligned with business goals and set in a way that it will return value in the shortest possible time.

Scope

Let’s start with the scope. From a technical perspective, data migration is the process of moving data from one system to another. While it might sound pretty straightforward, data migration is not simply copying the data. It is much more. To be able to define a realistic scope, you need to do a full assessment of your existing data warehouse and clearly define what must be migrated. You should document an as-is architecture and prepare a detailed list of:

• Databases, including all data warehouse staging areas
• Data model including all database objects such as tables and views and schemas
• ETL processes that populate and pull data in a variety of forms
• Orchestration processes
• Security schemas and authorization privileges
• Users and other data consumers
• BI tools and application

Don’t forget about business logic, data flows, and the dependencies. None of these steps must be neglected. The discovery phase is critical for project success. Project foundations are laid there. The more knowledge of how the current system works is gained the better.

Team

You must be aware that data migration is not only an IT project. It requires as much collaboration from the business side as it does from IT. Many migrations miss the desired outcome because the involvement of the business teams was underestimated. You must plan enough of their time to assist with migration. They know the data that is to be migrated better than anyone else.

Why is understanding the data crucial for project success? As already mentioned, data migration is usually accompanying a larger modernization project. Finally, it is not enough that data is just moved as-is. The bad news is that there are no shortcuts. Sooner or later, you will need to review it and structure it in a way that it will best fit into the new system.

Be prepared that during the migration all the traces of history will be discovered, and all the skeletons will come out of the closet. The more the existing system is customized and the longer it is already in use, the more attention is expected to be needed. Some of the data might need cleansing, some might need restructuring, and some will simply become obsolete and should be discarded. Don’t waste time and money on migrating everything. The one thing you don’t want in the new system, is the clutter. That is why it is very common that a data quality project runs in parallel. This is what takes up the most time and team effort and in order to be done well the data owners must dig in. They are also usually the ones who prepare the specifications, provide various definitions or explanations and do user acceptance tests.

Another important role that should not be missed is the project manager. Migrations are no different from other IT projects. The project manager is responsible for project coordination, communication and to ensure that scheduled duties are performed, and decisions reached within the agreed deadlines.

Don’t forget the system administrator who will take care of the infrastructure, user accounts, permissions, access rights, etc. The latter is undoubtedly much easier to handle for those of you deciding to migrate to Snowflake Data Cloud since you can forget about that additional work and costs when deploying and running a warehouse. Namely, with Snowflake you have no hardware to install and no operating system or database software to deploy. There are no patches to apply or database upgrades to schedule. This means near zero-management when setting up, the same goes for administration and operational costs. This baby runs by itself.

As you can see, numerous important activities can be accomplished only by in-house resources. Besides, migrations are not something that organizations do frequently, so, usually they involve external professionals. The reasons are many, from having limited internal resources at their disposal or lacking subject matter expertise to not knowing the target system well enough or not possessing the tools that would automate and accelerate the migration process.

Architecture and the Roadmap

Data migration usually accompanies a larger modernization project. Once you know the scope and you have a vision of the future system, it is time to define the migration roadmap. With large-scale migrations you should avoid a Big Bang approach if possible. Instead of only one and late go-live, you can mitigate the risk and show value sooner by breaking the scope into several independent parts that can be migrated and released in several stages. In such a situation make sure to define not only the to-be architecture but also the temporary ones in accordance with individual phases.

Snowflake’s advantage is its modern architecture. Its uniqueness comes from the fact that it was built from the ground up for the cloud and as such, it offers a much more sensible approach to architecting your data environment in the cloud. Here you can simplify your architecture and minimize the tools needed to run the processes. Because you can have your data lake alongside your data warehouse, you can finally store and manage all your data in one place, no matter if those are structured or semi-structured. This kind of set up also allows us to start small and build as we go, because we can add resources on the way of development and there’s no need for huge initial investments. Even further along, Snowflake’s pricing is competitively set – something you can make sure of here.

Regardless of the selected approach, it is essential for success to have a clearly defined migration process with all the activities and a detailed checklist, from data discovery and architecture design to migration itself, testing, go-live strategy and post-go-live activities, supported by a strong project management.

Having covered all the topics above, you are at the point where you can set realistic expectations about the project time-frame and budget. This is your chance to get among the highly desirable 17%* where successful migrations happen.

Katja Jurkovšek, Senior Consultant

*According to Gartner, 83% of all data migration projects either fail or exceed their budgets and timelines.

If you need a helping hand that will get you through, don’t hesitate.