Data in SQL Server - in the past

In the past, to process large amounts of data in SQL Server, you had to use an appliance called ADW (Analytics Data Warehouse) or also commonly known as PDW (Performance Data warehouse). ADW is not only a special version of SQL Server but a whole appliance including CPUs, memory, and storage. ADW was very expensive and because of the cost, wasn't used all that much. Even though it is expensive it is also very powerful. The reason behind that it was so powerful is it uses MPP. MPP is Massive Parallel Processing.  It divides any computing needs over multiple processing nodes with highly partitioned data.

SQL Server 2016

To do the same sorts of workloads that ADW accomplishes, you can actually accomplish that without all the limitations and expenses that occurred in ADW. The way to do this is to enable Polybase in SQL Server 2016. While you will need the Enterprise Edition of SQL Server to do this, it is much cheaper and easier than you probably imagine. At its cores, Polybase is a SQL Server implementation of Hive over HDFS (Hadoop). If you are familiar with Hadoop and Hive, you know that the power of Hadoop is in its distributed file system and map-reduce over multiple processing nodes. If you are familiar with Hive, you know it provides an SQL interface to produce map-reduce jobs over your Hadoop cluster. Now imagine combining HDFS, Map Reduce, and TSQL together. That is exactly what Polybase does under SQL Server.

With Polybase, you get a massively scalable and powerful MPP engine for your Data Analytic needs in a familiar and easy to use SQL Server implementation. If you need more power, you can just add more nodes to your cluster. If you need the benefits of relational technology it is there too. Just think of many things you can accomplish to processing tons and tons of data for your Data Warehousing and Analytic needs. The possibilities are endless. If you would like to know more about Polybase and how to architect a great powerful analytics solution, please feel free to contact me.