Hortonworks accelerates Spark at scale for the enterprise

“We continue to see customers across all industries derive real value from using Spark with Hortonworks Data Platform,” said Tim Hall, vice president of product management at Hortonworks. “Our customers rely on us to guide them on their Spark journey, and our ability to scale Spark against massive data-sets is remarkable.  With the inclusion of Spark 1.5.2 on HDP, customers can get new Spark capabilities and maximize its value for the enterprise.”

 

“Webtrends is working with Hortonworks to take Spark, Hive and Hadoop and execute these jobs in parallel,” said Peter Crossley, director of architecture, Webtrends. “This capability is critical because it allows us to combine the power of Big Data with the speed and flexibility of an ad hoc system. This means marketers will be able to ask any question of their unlimited data, no matter how structured it may be.”

 

Accelerating Apache Spark for Enterprise Scale  

Hortonworks is providing customers the easiest path for adopting Spark with Hadoop and allowing for innovation at scale. Customers can deploy modern, Spark-based applications alongside Hadoop workloads in a consistent, predictable and reliable way. In order to meet the requirements of enterprise customers, Hortonworks’ three main areas of focus for Spark include: 

 

Data Science Acceleration

• Improving data science productivity by enhancing Apache Zeppelin, currently available as a technical preview, and by contributing additional Spark algorithms and packages to ease the development of key solutions. One example is Project Magellan, an open source library for geospatial analytics that facilitates geospatial queries and builds upon Spark to solve hard problems dealing with geospatial data at scale.

 

Seamless Data Access

• Hortonworks is improving Spark’s integration with YARN, HDFS, Hive, HBase and ORC because customers are running Spark on YARN in combination and in conjunction with many of the other popular data access engines. Specifically, Hortonworks is working to further optimize data access via the new Data Source API. This will allow Spark SQL users to take full advantage of the following capabilities:

o   ORC File instantiation as a table

o   Column pruning

o   Language integrated queries

o   Predicate pushdown

 

Innovation at the Core

• Enhancing Spark’s enterprise security, governance, operations and overall readiness for real-world production deployment.

 

Fostering Community Innovation

Hortonworks has launched Hortonworks Community Connection (HCC), a new online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub. This new online community is an extension of Hortonworks’ open source roots and underscores its commitment to engaging with the community, and fostering community innovation and knowledge. More information about Project Magellan can be found within HCC.