Pinot (data store)

Pinot

Original author(s)|

* Kishore Gopalakrishna
* Xiang Fu

Developer(s)| Apache Pinot
Stable release|

0.7.1 / 18 March 2021; 5 years ago (2021-03-18)

Repository| Pinot repository
Written in| Java
Operating system| Cross-platform
Type|

* distributed
* real-time
* column-oriented data store

License| Apache License 2.0
Website| pinot.apache.org

Pinot is a column-oriented, open-source, distributed data store written in Java. Pinot is designed to execute OLAP queries with low latency[1]. It is suited in contexts where fast analytics, such as aggregations, are needed on immutable data, possibly, with real-time data ingestion.[2] The name Pinot comes from the Pinot grape vines that are pressed into liquid that is used to produce a variety of different wines. The founders of the database chose the name as a metaphor for analyzing vast quantities of data from a variety of different file formats or streaming data sources.[3]

Pinot was first created at LinkedIn after the engineering staff determined that there were no off the shelf solutions that met the social networking site's requirements like predictable low latency, data freshness in seconds, fault tolerance and scalability.[3][4] Pinot is used in production by technology companies such as Uber,[5] Microsoft,[2] and Factual[6].

## Contents

* 1 History
* 2 Architecture
* 2.1 Query management
* 2.2 Cluster management
* 3 Features
* 4 See also
* 5 References
* 6 External links

## History

Pinot was started as an internal project at LinkedIn in 2013 to power a variety of user-facing and business-facing products.The first analytics product at LinkedIn to use Pinot was a redesign of the social networking site's feature that allows members to see who has viewed their profile in real-time. The project was open-sourced in June 2015 under an Apache 2.0 license and was donated to the Apache Software Foundation by LinkedIn in June 2019.[3][2]

## Architecture

Architecture diagram of Apache Pinot

Pinot uses Apache Helix for cluster management. Helix is embedded as an agent within the different components and uses Apache ZooKeeper for coordination and maintaining the overall cluster state and health. All Pinot servers and brokers are managed by Helix. Helix is a generic cluster management framework to manage partitions and replicas in a distributed system.

### Query management

Queries are received by brokers—which checks the request against the segment-to-server routing table—scattering the request between real-time and offline servers.

### Cluster management

Pinot leverages Apache Helix for cluster management. Helix is a cluster management framework to manage replicated, partitioned resources in a distributed system. Helix uses Zookeeper to store cluster state and metadata.

## Features

Pinot shares similar features with comparable OLAP datastores, such as Apache Druid. Like Druid, Pinot is a column-oriented database with various compression schemes such as Run Length and Fixed Bit Length. Pinot supports pluggable indexing technologies \- Sorted Index, Bitmap Index, Inverted Index, Star-Tree Index, and Range Index, which are what primarily differentiates Pinot from other OLAP datastores.

Pinot supports near real-time ingestion from streams such as Kafka, AWS Kinesis and batch ingestion from sources such as Hadoop, S3, Azure, GCS. Like mostly, all other OLAP datastores and data warehousing solutions, Pinot supports a SQL-like query language that supports selection, aggregation, filtering, group by, order by, distinct queries on data

## See also

* List of column-oriented DBMSes

## References

1. ↑ Im, Jean-François; Gopalakrishna, Kishore; Subramaniam, Subbu; Shrivastava, Mayank; Tumbde, Adwait; Jiang, Xiaotian; Dai, Jennifer; Lee, Seunghyun et al. (2018-05-27). Pinot: Realtime OLAP for 530 Million Users. Sigmod '18. Association for Computing Machinery. pp. 583–594. doi:10.1145/3183713.3190661. ISBN 9781450347037. https://dl.acm.org/doi/10.1145/3183713.3190661#d13801648e1.
2. ↑ 2.0 2.1 2.2 Pawar, Neha. "Pinot Joins Apache Incubator" , LinkedIn Engineering, 01 April 2019
3. ↑ 3.0 3.1 3.2 Gopalakrishna, Kishore. "Open Sourcing Pinot: Scaling the Wall of Real-Time Analytics" (in en). LinkedIn. Archived from the original on 10 September 2015. https://web.archive.org/web/20150910081445/http://engineering.linkedin.com/pinot/open-sourcing-pinot-scaling-wall-real-time-analytics. Retrieved 3 September 2020.
4. ↑ Yegulalp, Serdar (2015-06-11). "LinkedIn fills another SQL-on-Hadoop niche" (in en). InfoWorld. https://www.infoworld.com/article/2934506/linkedins-pinot-fills-another-sql-on-hadoop-niche.html.
5. ↑ Wang, Haibo (15 January 2020). "Engineering SQL Support on Apache Pinot at Uber". Uber. https://eng.uber.com/engineering-sql-support-on-apache-pinot/. Retrieved 3 September 2020.
6. ↑ Melz, Eric (17 May 2020). "Pinot @ Factual" (in en). Archived from the original on 3 September 2020. https://web.archive.org/web/20200903081722/https://medium.com/@ericmelz/pinot-factual-6118ebaf64b3. Retrieved 3 September 2020.

## External links

* Official website

* v
* t
* e

Apache Software Foundation

Top-level
projects|

* Accumulo
* ActiveMQ
* Airflow
* Ambari
* Ant
* Apex
* Aries
* Apache HTTP Server
* APR
* Avro
* Axis
* Axis2
* Beam
* Bloodhound
* Brooklyn
* Buildr
* Calcite
* Camel
* CarbonData
* Cassandra
* Cayenne
* Chemistry
* CloudStack
* Cocoon
* Cordova
* CouchDB
* cTAKES
* CXF
* Derby
* Directory
* Drill
* Empire-db
* Felix
* Flex
* Flink
* Flume
* Forrest
* Geronimo
* Giraph
* Gump
* Hadoop
* Hama
* HBase
* Helix
* Hive
* Impala
* Jackrabbit
* James
* Jini
* JMeter
* Kafka
* Karaf
* Kudu
* Kylin
* Lucene
* Mahout
* Marmotta
* Maven
* MINA
* mod perl
* MyFaces
* NetBeans
* Nutch
* ODE
* OFBiz
* Oozie
* OpenEJB
* OpenJPA
* OpenNLP
* OрenOffice
* ORC
* PDFBox
* Parquet
* Phoenix
* POI
* Pig
* Pivot
* Qpid
* Roller
* RocketMQ
* Samza
* ServiceMix
* Shiro
* SINGA
* Sling
* Solr
* Spark
* Stanbol
* Storm
* SpamAssassin
* Sqoop
* Struts 1
* Struts 2
* Subversion
* SystemML
* Tapestry
* Thrift
* Tika
* Tomcat
* Trafodion
* Traffic Server
* UIMA
* Velocity
* Wicket
* Xalan
* Xerces
* Yetus
* ZooKeeper

Commons|

* BCEL
* BSF
* Daemon
* Jelly
* Logging

Incubator|

* MXNet
* Taverna
* XAP

Other projects|

* Batik
* Chainsaw
* FOP
* Ivy
* Log4j

Attic|

* Abdera
* AxKit
* Beehive
* Bluesky
* iBATIS
* Cactus
* Click
* Continuum
* Deltacloud
* Etch
* Excalibur
* Harmony
* HiveMind
* Jakarta
* Lenya
* Shale
* Shindig
* Slide
* stdcxx
* Tuscany
* Wave
* Wink
* XMLBeans

Licenses|

* Apache License

* Category

0.00

(0 votes)

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Software:Pinot_(data_store)&oldid=555934"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template