Apache spark to power microsoft bigdata analytics

Microsoft is  making a serious commitment to the open source Apache Spark cluster computing framework.

After dipping its toes into the Spark ecosystem last year, the company today launched a number of Spark-based services out of preview and announced that the on-premises version of R Server for Hadoop (which uses the increasingly popular open source R language for big data analytics and modeling) is now powered by Spark.
In addition, Microsoft announced that R Server for HDInsight (essentially the cloud-based version of R Server) is coming out of preview later this summer and Spark for Azure HDInsight is now generally available with support for managed Spark services from Hortonworks. Power BI, Microsoft’s suite of business intelligence tools, will now also support Spark Streaming to allow users to push real-time data from Spark right into Power BI.

Microsoft Shops prepare for next generation features in SQL Server 2016

Microsoft announces SQL Server 2016, an intelligent platform for a mobile first, cloud first world.  The next major release of Microsoft’s flagship database and analytics platform provides breakthrough performance for mission critical applications and deeper insights on your data across on-premises and cloud. Top capabilities for the release include: Always Encrypted – a new capability that protects data at rest and in motion, Stretch Database – new technology that lets you dynamically stretch your warm and cold transactional data to Microsoft Azure, enhancements to our industry-leading in-memory technologies for real-time analytics on top of breakthrough transactional performance and new in-database analytics with R integration.

Always Encrypted

Data security is top of mind, especially for mission critical applications, and SQL Server has been the enterprise database with the fewest security vulnerabilities six years running.*  To help customers with data security and compliance when using SQL Server on-premises or in the cloud, we are introducing Always Encrypted. Always Encrypted, based on technology from Microsoft Research, protects data at rest and in motion. With Always Encrypted, SQL Server can perform operations on encrypted data and best of all, the encryption key resides with the application in the customers trusted environment. Encryption and decryption of data happens transparently inside the application which minimizes the changes that have to be made to existing applications.

Stretch Database

Today, in the Ignite keynote, we showcased how you can gain the benefits of hyper-scale cloud in the box with new hybrid scenarios including Stretch Database. As core transactional tables grow in size, you may need to archive historical data to lower cost and to maintain fast performance. This unique technology allows you to dynamically stretch your warm and cold transactional data to Microsoft Azure, so your operational data is always at hand, no matter the size, and you benefit from the low cost of using Microsoft Azure.  You can use Always Encrypted with Stretch Database to extend your data in a more secure manner for greater peace of mind.

Real-time Operational Analytics & In-Memory OLTP

Building on our industry leading and proven in-memory technologies, customers will benefit from the combination of real-time operational analytics with blazing fast transactional performance – a first among enterprise vendors.  For In-Memory OLTP, which customers today are using for up to 30x faster transactions than disk based systems, you will now be able to apply this technology tuned for transactional performance to a significantly greater number of applications as well as benefit from increased concurrency.  With these enhancements, we also introduce the unique capability to use our in-memory columnstore delivering 100X faster queries with in-memory OLTP for in-memory performance and real-time operational analytics.

Built-in Advanced Analytics, PolyBase and Mobile BI

For deeper insights into data, SQL Server 2016 expands its scope beyond transaction processing, data warehousing and business intelligence to deliver advanced analytics as an additional workload in SQL Server with proven technology from Revolution Analytics.  We want to make advanced analytics more accessible and increase performance for your advanced analytic workloads by bringing R processing closer to the data and building advanced analytic capabilities right into SQL Server.  Additionally, we are building PolyBase into SQL Server, expanding the power to extract value from unstructured and structured data using your existing T-SQL skills. With this wave, you can then gain faster insights through rich visualizations on many devices including mobile applications on Windows, iOS and Android.

Additional capabilities in SQL Server 2016 include:

  • Additional security enhancements for Row-level Security and Dynamic Data Masking to round out our security investments with Always Encrypted.
  • Improvements to AlwaysOn for more robust availability and disaster recovery with multiple synchronous replicas and secondary load balancing.
  • Native JSON support to offer better performance and support for your many types of your data.
  • SQL Server Enterprise Information Management (EIM) tools and Analysis Services get an upgrade in performance, usability and scalability.
  • Faster hybrid backups, high availability and disaster recovery scenarios to backup and restore your on-premises databases to Azure and place your SQL Server AlwaysOn secondaries in Azure.

In addition, there are many more capabilities coming with SQL Server 2016 that deliver mission critical performance, deeper insights on your data and allow you to reap the benefits of hyper-scale cloud.

Last week at Build we announced exciting innovations to support our mission of making it easier to work with your data, no matter how big or complex.  We also shared how we are bringing capabilities to the cloud first in Azure SQL Database as with such as Row-level security and Dynamic Data Masking and then bringing the capabilities, as well as the learnings from running these at hyper-scale, back to SQL Server to improve our on-premises offering.  Thus, all our customers benefit from our investments and learnings in Microsoft Azure.  In addition to our hybrid cloud scenarios and investments in running SQL Server 2016 in Azure Virtual Machine, SQL Server delivers a complete database platform for hybrid cloud, enabling you to more easily build, deploy and manage solutions that span on-premises and cloud.

As the foundation of our end-to-end data platform, with this release of SQL Server we continue to make it easier for customers to maximize your data dividends. With SQL Server 2016 you can capture, transform, and analyze any data, of any size, at any scale, in its native format —using the tools, languages and frameworks you know and want in a trusted environment on-premises and in the cloud.

Be sure to visit the SQL Server 2016 preview page to read about the capabilities of SQL Server 2016 and sign-up to be notified once the public preview is available.

Cheap , Easy and Fast DataWarehousing – Our Implementation of Amazon Redshift

Conventional methods were proven either slow and/or expensive in providing a complete Analytics Data Warehousing/Reporting solution to one of our clients who provide world-wide IPTV Services.     We then decided to scrap plans for a conventional data warehouse after doing preliminary POC Testing on Redshift.
 
With 1/10th of the cost and high performance and ease of manageability and moving away from traditional in house IT management and support,  the extensively scalable cloud based solution seem to be a perfect fit.      Redshift’s capability to work in sync with Hadoop and integrations with Amazon’s Elastic Map Reduce was another great motivation factor in selection of the technology.
 
There was never any worries about capacity that redshift can handle or other multi-tenancy or scalability factors.   The other primary factor was also being able to choose a feasible, flexible and ease of use reporting system that would work with Redshift and also have the capability of delivering ad-hoc reporting features with multi-tenancy support.
 
The estimated size of the database is expected to be several 10s of 100s of Terrabytes across multiple tenants all residing in a single redshift cluster scalable with multiple compute nodes.   The easy of  management without indexes and just having control over the sort and distribution keys was a significant time saver in data management.  Within few seconds the system is capable of delivering reports of various time granularity shifting through several years of data.

We at Applayatech, provide Redshift consulting service, which includes Redshift data warehouse service too at worthwhile Amazon Redshift pricing.

Contact us for Redshift consulting service, we also provide Redshift data warehouse service. Call us to know more about Amazon Redshift pricing today.