Argonne National Laboratory

Analyzing the Impact of Supporting Out-of-Order Communication on In-order Performance with iWARP

TitleAnalyzing the Impact of Supporting Out-of-Order Communication on In-order Performance with iWARP
Publication TypeConference Paper
Year of Publication2007
AuthorsBalaji, P, Feng, W, Bhagvat, S, Panda, DK, Thakur, R, Gropp, WD
Conference NameIEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC)
Date Published10/2007
Conference LocationReno, Nevada
Other NumbersANL/MCS-P1448-0707

Due to the growing need to tolerate network faults and congestion in high-end computing systems, supporting multiple network co9mmunication paths is becoming increasingly important. However, multi-path communication comes with the disadvantage of having to deal with the out-of-order arrival of packets (because packets may traverse different paths). While modern networking stacks such as the Internet Wide-Area RDMA Protocol (iWARP) over 10-Gigabit Ethernet (10GE) support multi-path communication, they do not handle out-of-order packets primarily owing to the overhead on in-order communication that it adds. Specifically, in iWARP, supporting out-of-order packets requires every packet to carry additional information causing significant overhead on packets that arrive in-order. Thus, in this paper, we analyze the trade-offs in designing a feature-complete iWARP stack, i.e., one that provides support for out-of-order arriving packets, and thus, multi-path systems, while focusing on the performance of in-order communication. We propose three feature-complete designs of iWARP and analyze the pros and cons of each of these designs using performance experiments based on several micro-benchmarks as well as an iso-surface visual rendering application. Our analysis reveals with the iWARP design providing the best overall performance depends on the particular characteristics of the upper layers and that different designs are optimal based on the metric of interest.