Qubole

Overview

Qubole is an enterprise cloud data lake platform designed to simplify big data engineering, ad-hoc analytics, and machine learning at scale. Built to manage processing frameworks like Apache Spark, Presto, and Hive, Qubole automates cluster management, auto-scaling, and cloud resource optimization. The system connects multi-cloud storage environments to reduce infrastructure spending and administrative complexity. It empowers data teams to run high-concurrency workloads seamlessly across AWS, Azure, and Google Cloud.

What it does

• Automates cloud cluster creation, auto-scaling, and lifecycle management for big data frameworks.
• Optimizes query workloads across Apache Spark, Hive, Presto, and Airflow processing engines.
• Provides real-time financial controls and cost attribution across cloud data lake environments.
• Enables self-service data engineering, ad-hoc querying, and machine learning model execution.

Who it's for

• Data Engineers
• Data Scientists
• Enterprise Analytics Directors
• Cloud Infrastructure Leads

Why it works

• Autonomous auto-scaling dynamically adjusts cloud compute capacity based on workload demand.
• Cross-cloud support unifies operations across AWS, Microsoft Azure, and Google Cloud Platform.
• Native integration with open-source engines like Spark and Presto prevents vendor lock-in.
• Automated cluster lifecycle management lowers compute costs through aggressive spot instance usage.

Founder story

Qubole was founded in 2011 in Santa Clara, California, by Ashish Thusoo and Joydeep Sen Sarma. While creating the data platform team at Facebook and co-creating Apache Hive, the founders experienced firsthand the immense friction of managing massive cloud data infrastructure. They built Qubole to automate cloud data lake operations and make big data self-service for enterprise teams. Qubole was acquired by Idera, Inc. in 2020.

Alternatives

• Databricks
• Snowflake
• Cloudera
• Amazon EMR

Comparison overview

• Qubole emphasizes autonomous multi-engine cluster management and cloud spot instance optimization.
• Unlike closed data warehouses such as Snowflake, Qubole operates directly on top of open data lakes utilizing managed open-source engines like Spark, Presto, and Hive.