Building a scalable distributed data platform using lambda architecture

K-REx Repository

Show simple item record

dc.contributor.author Mehta, Dhananjay
dc.date.accessioned 2017-04-18T13:21:39Z
dc.date.available 2017-04-18T13:21:39Z
dc.date.issued 2017-05-01 en_US
dc.identifier.uri http://hdl.handle.net/2097/35403
dc.description.abstract Data is generated all the time over Internet, systems sensors and mobile devices around us this is often referred to as ‘big data’. Tapping this data is a challenge to organizations because of the nature of data i.e. velocity, volume and variety. What make handling this data a challenge? This is because traditional data platforms have been built around relational database management systems coupled with enterprise data warehouses. Legacy infrastructure is either technically incapable to scale to big data or financially infeasible. Now the question arises, how to build a system to handle the challenges of big data and cater needs of an organization? The answer is Lambda Architecture. Lambda Architecture (LA) is a generic term that is used for scalable and fault-tolerant data processing architecture that ensures real-time processing with low latency. LA provides a general strategy to knit together all necessary tools for building a data pipeline for real-time processing of big data. LA comprise of three layers – Batch Layer, responsible for bulk data processing, Speed Layer, responsible for real-time processing of data streams and Service Layer, responsible for serving queries from end users. This project draw analogy between modern data platforms and traditional supply chain management to lay down principles for building a big data platform and show how major challenges with building a data platforms can be mitigated. This project constructs an end to end data pipeline for ingestion, organization, and processing of data and demonstrates how any organization can build a low cost distributed data platform using Lambda Architecture.
dc.language.iso en_US en_US
dc.publisher Kansas State University en
dc.subject Big data en_US
dc.subject Hadoop en_US
dc.subject Data supply chain en_US
dc.subject Spark en_US
dc.subject Map Reduce en_US
dc.subject Lambda architecture en_US
dc.title Building a scalable distributed data platform using lambda architecture en_US
dc.type Report en_US
dc.description.degree Master of Science en_US
dc.description.level Masters en_US
dc.description.department Department of Computer Science en_US
dc.description.advisor William H. Hsu en_US
dc.date.published 2017 en_US
dc.date.graduationmonth May en_US


Files in this item

This item appears in the following Collection(s)

Show simple item record

Search K-REx


Advanced Search

Browse

My Account

Statistics








Center for the

Advancement of Digital

Scholarship

cads@k-state.edu