Unlock hidden savings in your Snowflake environment. This guide reveals proven strategies to reduce compute and storage costs. Optimise your warehouses, monitor usage, and gain full control over your Snowflake costs.
Pay as you go? But what are you really paying for?
Pay as you go models are ever growing, ultimately giving you more control over your costs. Snowflake operates an easy-to-understand costing model, with any additional Snowflake managed features easy to grasp. There are two main groups of cost; storage, and compute.
Snowflake Storage Costs
When considering the storage cost model in Snowflake, it's important to note that the pricing is based on the average number of terabytes stored throughout the month. For example, at the time of publishing this blog, for Snowflake running on Amazon Web Services (AWS) in London, Snowflake will charge you $42.00 per TB for on demand pricing, however, you can cut this down to $24 for any upfront payments. This can be a real advantage if you know how much data you must store or are looking to store in the near future.
To provide context, let us compare the storage costs of the three major cloud providers: Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure (Azure):
- AWS is approximately $23.55 per TB a month.
- GCP is also in the same region, at $23.00 per TB a month.
- Azure is the cheapest at $19.00 per TB a month.
All the above exclude any free tiers of storage. If the required storage capacity is known in advance, it is generally more cost-effective to pay upfront, as this aligns with the pricing structure of the main hyperscalers. It is also worth noting that Snowflake offers varying rates depending on the chosen cloud provider and the specific region where the Snowflake environment is hosted
Snowflake Compute Costs
Snowflake have their own compute currency called ‘credits’. A credit is worth a value based on the Snowflake tier you require. If we look at AWS in London, then for the Standard version of Snowflake it will cost you $2.70 per credit. If you want the Enterprise level features it jumps to $4.00 per credit, then up to $5.40 per credit for the Business Critical version that includes key features for data compliance. The main question here is, what version do I really need?
To answer that question, you really need to know the difference between the versions of Snowflake. Enterprise provides some great additional features such as tagging, materialised views, row and column level data obfuscation, up to 90 days’ time travel on tables, multi-cluster ability, query acceleration and much more. But, do you really need these for your use case? The full edition feature matrix can be found here.
The next question is, what uses compute credits? It's simple as it is based on the size and time you are running Snowflake compute warehouses (also known as virtual warehouses). There are various levels of warehouses, from XS (extra small), through to 6XL.
Figure 1 - A Cost table of warehouses size to credits per hours in Snowflake.
The pricing table for the size of warehouses you can spin up is consistent. Warehouses consume double the credits for the next size up starting at one credit per hour (just $2.70 per hour) but warehouses won’t be on 24/7 like an on-premises database, they are only on as and when you need them. (Note these are charged per cluster within the warehouse, by default it's only one cluster for standard accounts). Snowflake will charge you sixty seconds minimum for spinning up a warehouse no matter how much is calculated within that time. Check here for full details.
There are some additional costs for extra services that Snowflake manages via their compute resources. There are things like materialised views (Enterprise only) and Snowpipe to name a couple. They are charged at their own credit per hour rate, but only when being used.
It's as simple as that. You don’t have to worry about any management of resources. This is all Snowflake's responsibility. No need to have database administrators looking after on-premises servers, worrying about keeping it updated and secure, backups, maintenance etc as all of this is done for you, empowering you to focus on getting your data to insight as quickly as possible.
How can you monitor your Snowflake instance?
It can be tricky to remain on top of costs when processes get added, so keeping track of those and monitoring them is essential to making sure you don’t foot a heavy bill at the end of the month. Snowflake has some great in-house tools to prevent overspend by enabling the set up of resource monitors at both warehouse and account level to make sure you never spend more than your budget.
There is limited visual insights given by Snowflake, but a whole load of data within the account monitoring schemas that Snowflake provides for users to keep tabs on their account usage. There are various views that will tell you how many credits you are consuming, by query, tasks, pipelines but they will need some analytics skills to pluck out the views that will be useful to you. Something we have done at Ometis is take these data points and built them into an easy to consume Qlik SaaS dashboard to help with our analysis.
Additional cost optimisations for Snowflake
Spending too much? Well luckily there are some quick wins available to start reducing your costs in just a matter of minutes.
- Optimising your warehouse idle time: Since you are billed for a minimum of one minute, it makes sense to make sure your warehouse has a low idle time and closes as soon as possible after use. Making sure this is set ensures warehouses aren't left on whilst not processing anything. This can be achieved with a simple SQL query.
- Consolidating processes: Running multiple processes will naturally occur, but if teams are running in silos, they may run these processes at the same time or different times on different warehouses leading to more compute resources not being fully unlisted. Optimising these can lead to cost savings.
- Empowering your team: Educating about the cost of Snowflake to your team and users can have massive impact by making users more cost conscious. There is a great selection of self-paced content that Snowflake offers for free here.
- Pick the right size warehouse: More power can process your work quicker saving credits in some cases. Picking whether to scale up or out is of key importance. There are some common use cases (one mentioned below) to know when to use what size of warehouse.
- Removing Database objects: it sounds obvious, but that copy of a large table that’s gone unnoticed can start to cost you a chunk of change each month. Or a process running that’s not required any more, just shutting these down can have impact.
These are just some of the high-level things you can do to keep cost down and prevent it from getting out of hand. There are many more activities you can do to get your costs down to a minimum. Our Snowflake monitoring dashboard can give you spend on your processes. Request a free demo to see how it works!
At Ometis we currently run all eight of our production systems through Snowflake via various pipeline methods and they cost in total less than $2.00 per day on a standard Snowflake account. We have implemented all the above methods as well as others to keep our costs as low as possible.
Monitoring your costs is not a one-time activity, but rather an ongoing activity. For example a file that has landed might be incorrect and you don’t have the right error handling. That could cause a warehouse to be on much longer than you anticipated and being able to spot errors and cost changing over time gives your more understanding of costs to be able to make impact.
Time = Money
So, here’s a myth for you, “using a larger warehouse will always cost me more”. Incorrect! In fact, there are cases where using larger warehouses could reduce cost by saving on time. A simple example is as follows.
- Running one extra small warehouse for one hour to process one pipeline will cost you $2.70
- Running one medium warehouse for five minutes to process one pipeline will cost you $0.90
Figure 2 - A Picture to demonstrate that increasing warehouse size, doesn’t always increase cost.
Being able to process the data in a quicker time than consuming the higher rate of credits can really boost your savings.
This is not always the case, there are instances where you might need a large amount of parallel processing (file ingestion is a good example), whereas a single large and small warehouse will likely process the files in a similar time. So, using a multi cluster small warehouse to get the job done quickly with the smaller amount of credit usage is key.
Summary
This blog has delved into the world of snowflake pricing, empowering you to manage your costs effectively. Some of the key takeaways are:
- Pay as you go model: Only pay for what you use.
- Snowflake tier: Don’t need all the extra features? Use the standard version of and upgrade later.
- Optimisation tips: Implement some of the quick wins like warehouse idle time to changes overnight.
- Size Matters: Picking that right warehouse can be a game changer on costs.
- Monitor, Monitor, Monitor! Get a free demo of our cost monitor to see what insights you can get to save on your costs.
Comments