Hortonworks accelerates Spark at scale for the enterprise

Hortonworks has given details of coming advancements of Hortonworks Data Platform (HDP™) with the in-memory analytic capabilities of Spark. Apache Spark 1.5.2 will include support for Spark SQL and Spark Streaming. Hortonworks’ commitment to Spark is focused on helping customers accelerate data science, maintain seamless data access, drive innovation at the core and ultimately scale for the enterprise.

“We continue to see customers across all industries derive real value from using Spark with Hortonworks Data Platform,” said Tim Hall, vice president of product management at Hortonworks. “Our customers rely on us to guide them on their Spark journey, and our ability to scale Spark against massive data-sets is remarkable.  With the inclusion of Spark 1.5.2 on HDP, customers can get new Spark capabilities and maximize its value for the enterprise.”

 

“Webtrends is working with Hortonworks to take Spark, Hive and Hadoop and execute these jobs in parallel,” said Peter Crossley, director of architecture, Webtrends. “This capability is critical because it allows us to combine the power of Big Data with the speed and flexibility of an ad hoc system. This means marketers will be able to ask any question of their unlimited data, no matter how structured it may be.”

 

Accelerating Apache Spark for Enterprise Scale  

Hortonworks is providing customers the easiest path for adopting Spark with Hadoop and allowing for innovation at scale. Customers can deploy modern, Spark-based applications alongside Hadoop workloads in a consistent, predictable and reliable way. In order to meet the requirements of enterprise customers, Hortonworks’ three main areas of focus for Spark include: 

 

Data Science Acceleration

  • Improving data science productivity by enhancing Apache Zeppelin, currently available as a technical preview, and by contributing additional Spark algorithms and packages to ease the development of key solutions. One example is Project Magellan, an open source library for geospatial analytics that facilitates geospatial queries and builds upon Spark to solve hard problems dealing with geospatial data at scale.

 

Seamless Data Access

  • Hortonworks is improving Spark’s integration with YARN, HDFS, Hive, HBase and ORC because customers are running Spark on YARN in combination and in conjunction with many of the other popular data access engines. Specifically, Hortonworks is working to further optimize data access via the new Data Source API. This will allow Spark SQL users to take full advantage of the following capabilities:

o   ORC File instantiation as a table

o   Column pruning

o   Language integrated queries

o   Predicate pushdown

 

Innovation at the Core

  • Enhancing Spark’s enterprise security, governance, operations and overall readiness for real-world production deployment.

 

Fostering Community Innovation

Hortonworks has launched Hortonworks Community Connection (HCC), a new online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub. This new online community is an extension of Hortonworks’ open source roots and underscores its commitment to engaging with the community, and fostering community innovation and knowledge. More information about Project Magellan can be found within HCC.

Geotab marks a quarter-century of success, driven by innovation and collaboration, evolving into a...
SNIA launches Storage.AI to tackle AI data challenges through collaborative, industry-standard...
Baker Hughes introduces its Cordant™ Asset Strategy Data Services on SAP® Store, integrating...
Litera unveils cutting-edge updates to its Foundation Platform, integrating AI-powered tools into...
A survey by Storyblok highlights how AI and other factors influence developers' productivity and...
IFS introduces a new Emissions Management module in partnership with Climatiq to embed...
ACTFORE secures a pioneering patent in the data mining field, revolutionising breach response with...
Confluent announces a $200 million investment to enhance its partner ecosystem, driving innovation...