TCP testing: how well does ns2 match reality?

New transport protocols continue to appear as alternatives to the Transmission Control Protocol (TCP). Many of these are are designed to address TCP's inefficiency in operating over paths with a high bandwidth-delay product (BDP). To test these new protocols, especially comparatively, and to understand their interactions, extensions to the ns2 simulator allow real code from the linux kernel to be used within the ns2 simulations. However, how does the performance of such configurations compare to test-bed experiments of the same configuration? Although, anecdotally, there are often comments within the research community about such issues, there are no studies that quantify the differences for a specific protocol suite. Using a simple testbed, we assess four different transport protocols in a comparative study to examine how well ns2 matches reality. Our tests are all conducted at 100Mb/s over a wide range of delay and router buffer conditions: end-to-end delays from 25ms to 400ms, with end-to-end path buffering of 20% to 100% of the BDP. We find that in our simple configuration, there are significant differences in performance between ns2 and the testbed.