NDLI: Efficient Similarity Joins on Massive High-Dimensional Datasets Using MapReduce

Content Provider	IEEE Xplore Digital Library
Author	Wuman Luo Haoyu Tan Huajian Mao Ni, L.M.
Copyright Year	2012
Abstract	High-dimensional similarity join (HDSJ) is critical for many novel applications in the domain of mobile data management. Nowadays, performing HDSJs efficiently faces two challenges. First, the scale of datasets is increasing rapidly, making parallel computing on a scalable platform a must. Second, the dimensionality of the data can be up to hundreds or even thousands, which brings about the issue of dimensionality curse. In this paper, we address these challenges and study how to perform parallel HDSJs efficiently in the MapReduce paradigm. Particularly, we propose a cost model to demonstrate that it is important to take both communication and computation costs into account as dimensionality and data volume increases. To this end, we propose DAA (Dimension Aggregation Approximation), an efficient compression approach that can help significantly reduce both these costs when performing parallel HDSJs. Moreover, we design DAA-based parallel HDSJ algorithms which can scale up to massive data sizes and very high dimensionality. We perform extensive experiments using both synthetic and real datasets to evaluate the speedup and the scale up of our algorithms.
Starting Page	1
Ending Page	10
File Size	543749
Page Count	10
File Format	PDF
ISBN	9781467317962
e-ISBN	9780769547138
DOI	10.1109/MDM.2012.25
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2012-07-23
Publisher Place	India
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Vectors Approximation methods Approximation algorithms Algorithm design and analysis Time series analysis Computational modeling Data models
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Libarray of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Fuzzy Joins Using MapReduce

Parallel feature selection using positive approximation based on MapReduce

Distributed Diversification of Large Datasets

Intelligent Similarity Joins for Big Data Integration

MELODY-JOIN: Efficient Earth Mover's Distance similarity joins using MapReduce

Design and Implementation of Parallel Term Contribution Algorithm Based on Mapreduce Model

SE-CDA: A scalable and efficient community detection algorithm

SEJ: An Even Approach to Multiway Theta-Joins Using MapReduce

Approximate Search on Massive Spatiotemporal Datasets

Efficient Similarity Joins on Massive High-Dimensional Datasets Using MapReduce

Similar Documents

Fuzzy Joins Using MapReduce

Parallel feature selection using positive approximation based on MapReduce

Distributed Diversification of Large Datasets

Intelligent Similarity Joins for Big Data Integration

MELODY-JOIN: Efficient Earth Mover's Distance similarity joins using MapReduce

Design and Implementation of Parallel Term Contribution Algorithm Based on Mapreduce Model

SE-CDA: A scalable and efficient community detection algorithm

SEJ: An Even Approach to Multiway Theta-Joins Using MapReduce

Approximate Search on Massive Spatiotemporal Datasets

Efficient Similarity Joins on Massive High-Dimensional Datasets Using MapReduce