| Title: | OS5-2 Bandwidth-Aware Routing Mechanism to Control Hadoop Shuffle Traffic over SoftwareDefined Networking |
|---|---|
| Publication: | ICAROB2025 |
| Volume: | 30 |
| Pages: | 119-122 |
| ISSN: | 2188-7829 |
| DOI: | 10.5954/ICAROB.2025.OS5-2 |
| Author(s): | Ming-Syuan Wu, Cheng-Han Lin, Wen-Shyang Hwang, Ce-Kuen Shieh, Mao-Syun Lin |
| Publication Date: | February 13, 2025 |
| Keywords: | Software-Defined Network, Apache Hadoop, MapReduce, Shuffle |
| Abstract: | MapReduce is the main program in the Hadoop computing architecture. It involves mapping and reduction processes that require servers to exchange large amounts of data with each other. Data exchange between servers during the shuffle stage can lead to insufficient network bandwidth. In the present study, an algorithm for distributing Hadoop shuffle traffic to all possible paths was developed. Software-defined networking (SDN), a centrally controlled form of network architecture, was used to collect network status parameters to allocate shuffle traffic. During traffic distribution, the Mininet simulator was used to build the network topology and simulate operation. The Ryu controller was used as the SDN controller. The simulation results indicated that the proposed method was superior to the Spanning Tree Protocol (STP) and bandwidth-aware algorithm in reducing Hadoop completion time through effective shuffle traffic distribution. |
| PDF File: | https://alife-robotics.co.jp/members2025/icarob/data/html/data/OS/OS5/OS5-2.pdf |
| Copyright: | © The authors. This article is distributed under the terms of the Creative Commons Attribution License 4.0, which permits non-commercial use, distribution and reproduction in any medium, provided the original work is properly cited. See for details: https://creativecommons.org/licenses/by-nc/4.0/ |
(c)2008 Copyright The Regents of ALife Robotics Corporation Ltd. All Rights Reserved.