A uni-directional traceroute (run only in one direction) can be used to accurately spot a problem area in a route.
If one does see some issues in a traceroute – that start at one hop and carry on forward (e.g. packet loss, or latency past the expected given the distance traveled between the two hops3), then it is imperative to run it bi-directionally. What I mean by this is, run, if possible at the same time or as close together time-wise as possible, a route from the user host to the server, and from the server to the user host.
It is important to do this, as often routes are asymmetric (take one path in one direction, yet another path in the return direction), and one needs to see the route both ways to be able to more accurately pinpoint where a problem may lie.
For example, trace from a customer to their server (this example now also brings together all the matters discussed in this article):
Tracing route to cyberfret.com [209.103.172.27] over a maximum of 30 hops:
1 7 ms 7 ms 9 ms 10.33.96.1
2 7 ms 9 ms 17 ms gig2-1.ghnaoh1-ybr1.columbus.rr.com [24.95.81.225]
3 15 ms 9 ms 9 ms srp9-0.clmboh1-rtr2.columbus.rr.com [65.25.129.94]
4 22 ms 20 ms 20 ms pos14-0.clmboh1-rtr3.columbus.rr.com [65.25.128.138]
5 22 ms 20 ms 19 ms son0-1-2.mtgmoh1-rtr0.columbus.rr.com [65.25.128.229]
6 25 ms 24 ms 22 ms te-3-2.car1.Cincinnati1.Level3.net [4.78.216.13]
7 23 ms 31 ms 34 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.132.206]
8 22 ms 35 ms 35 ms ae-78.ebr3.Chicago1.Level3.net [4.69.134.62]
9 31 ms 33 ms 35 ms ae-2.ebr2.Washington1.Level3.net [4.69.132.70]
10 30 ms 31 ms 54 ms ae-82-82.csw3.Washington1.Level3.net [4.69.134.154]
11 * * * Request timed out.
12 38 ms 38 ms 40 ms ge6-2.core1.dca2.hopone.net [66.36.224.190]
13 * * * Request timed out.
14 34 ms 34 ms 32 ms client.covesoft.net [209.103.172.27]
And a trace from their server back to their user host:
traceroute to 24.95.81.225 (24.95.81.225), 64 hops max, 40 byte packets
1 c-vl101-d1.acc.dca2.hopone.net (66.36.230.2) 0.381 ms 0.322 ms 0.241 ms
2 gec2.core2.dca2.hopone.net (66.36.224.233) 0.213 ms 0.199 ms 0.167 ms
3 gi1-0-0.ar2.wdc2.gblx.net (66.36.224.169) 1.339 ms 1.406 ms 1.368 ms
4 te1-1-10g.ar3.DCA3.gblx.net (67.17.108.134) 2.887 ms 2.585 ms 2.523 ms
5 pop2-ash-S2-1-0.atdn.net (66.185.136.9) 2.488 ms 2.608 ms 2.444 ms
6 bb2-ash-P3-0.atdn.net (66.185.151.150) 1.878 ms 1.941 ms 1.894 ms
7 bb2-cin-P3-0.atdn.net (66.185.153.61) 22.294 ms 22.182 ms 22.302 ms
8 pop1-cin-P1-0.atdn.net (66.185.133.3) 22.655 ms 22.407 ms 22.752 ms
9 RR-Cincinnati.atdn.net (66.185.133.6) 23.632 ms 23.698 ms 23.783 ms
10 pos13-1.clmboh1-rtr2.columbus.rr.com (24.95.81.194) 27.110 ms 26.940 ms 26.829 ms
11 srp1-1.ghnaoh1-ybr1.columbus.rr.com (65.25.129.68) 27.991 ms * 28.186 ms
As clearly evident, the route here is asymmetrical (as is often the case, given the various different routing preferences among various providers (and multiple, often countless, routes from point A to point B), each selecting the “best” routes differently from others – many often do that based on cost vs. performance, though the hopone.net network we use always does everything 100% based on optimal performance: shortest path, lowest latency, and always no packet loss). The route in (from customer to their server) goes from Road Runner in Columbus, OH to Level 3 in Cincinnati, OH and from there to HopOne in McLean, VA (the Level 3 [Old Meadow Road] McLean, VA site is misleadingly called Washington.Level3.net). The route out (from server to customer host), however, goes from HopOne in McLean, VA to GlobalCrossing in Washington, DC, then to AOL in Ashburn, VA and then to Road Runner in Cincinnati, OH, from where it travels to the end host in Columbus, OH.
The latency on the out route is lower, as it is tracing to the first real (not 10.x.x.x or 192.x.x.x non-routable IPs, that are not routable, private IP space used by various networks) hop in the customer’s in route, since the customer did not provide us with their actual host IP. As is evident in the in route, the customer has 7ms latency from their cable modem to their first Internet hop. As the out route traces to their gateway, it is, as expected, 7ms lower, as it does not trace all the way to the customer’s end host that is 7ms further away.
The in route shows two hops as “* * * Request timed out.” that, as earlier explained in this article, are no reason for concern, as long as the end host is reached with no problems. One of the hops, hop 11, is a Level 3 router that has been configured not to respond to traceroutes or pings (most likely for security and/or performance reasons), while the other hop, hop 13, is a HopOne distribution switch that is in non-routable IP space – which typically will show up in normal traceroutes (as hop 12 does), but in this case does not. However, as the end host is reached and shows no packet loss (3/3 pings to it have arrived perfectly fine, and all with consistent latency), it is evident that no problems are present. To ensure, however, the customer in this case would be suggested to run a recursive ping (ping –t) a 100+ times from their end host to the server, to verify that there are indeed no issues (there should not be, as the routes both ways are as close to perfect as possible, with stable latencies and no real packet loss anywhere). If there are any issues, such would be most likely present on the Road Runner end in Columbus, OH, as in the outgoing trace there are some packets lost there at the customer’s end gateway hop (however 1/3 packet loss does not have enough samples to go by to be objective, and it most likely is not reflective of 33% packet loss on that end host – as it is a router, it likely treats ping with the lowest priority and has a maximum of ping requests responses per second configured, so while it may be an indication of an issue there, it may be also simply the router doing what it has been configured to do and reject frequent pings).
/mtr –report-cycles=30 –report extby.neolocation.net
HOST: salviori.us.ded.neolocation Loss% Snt Last Avg Best Wrst StDev
1. h-vl202-d3.acc.dca2.hopone.n 0.0% 30 0.4 0.6 0.4 2.6 0.6
2. gec3.core2.dca2.hopone.net 0.0% 30 0.4 0.5 0.4 2.2 0.4
3. ge6-0.core1.dca2.hopone.net 0.0% 30 0.4 0.5 0.3 3.5 0.6
4. cw.peer.nyc2.hopone.net 0.0% 30 5.9 11.2 5.6 117.3 21.3
5. so-3-0-0-dcr1.nyk.cw.net 0.0% 30 5.8 13.0 5.7 121.9 27.2
6. so-7-0-0-dcr1.was.cw.net 0.0% 30 6.2 11.7 6.1 98.9 19.0
7. so-0-0-0-dcr1.par.cw.net 0.0% 30 86.4 91.2 86.3 136.4 11.7
8. so-7-0-0-dcr1.fra.cw.net 0.0% 30 103.4 97.0 94.9 124.1 5.5
9. so-4-0-0-zar1.fri.cw.net 0.0% 30 95.1 96.1 94.9 120.0 4.5
10. romteleco-gw.fri.cw.net 50.0% 30 455.1 462.5 450.4 532.3 23.3
11. spb-dsr2-ge0-2-0-0.rt-comm.r 40.0% 30 488.0 492.3 484.1 568.6 19.1
12. 195.161.4.34 26.7% 30 486.7 487.0 484.2 491.4 1.7
13. J10-1-B57.ge-1-0-0-7.peterst 56.7% 30 488.1 487.4 485.3 488.8 1.2
14. BELPAK.peterstar.net 60.0% 30 498.7 498.3 494.5 501.6 1.9
15. 193.232.248.110 56.7% 30 497.8 519.5 494.1 778.8 77.9
16. 193.232.249.185 53.3% 30 499.9 499.2 494.5 503.3 2.7
17. 86.57.250.98 63.3% 30 498.0 500.6 495.3 521.4 7.4
In this route, it is evident in the highlighted section, that there is an issue with “romteleco” (rt-comm.ru) gateway to their IP transit provider, Cable & Wireless (cw.net) – it would seem that this their gateway to C&W is seriously overloaded/congested, as the high packet loss carries all the way through past it on every hop. At such high packet loss (27-60%), the connectivity is likely to be severely limited.
In this particular case, an incoming route (from the customer to their server) would likely show the opposite: packet loss always after the “romteleco” gateway with C&W and onwards from there, as that is where the issue is caused. Seeing such an incoming route to compare it with the outgoing route would then confirm where the problem lies, if in both direction of routes it starts happening after one particular hop (in this case, the “romteleco” link to C&W).
Understanding Traceroute: A Four-part Series
Part 1: Traceroute can be used to measure performance
Part 2: Traceroute/tracert-like applications or commands will provide accurate results
Part 3: * * * s (apparent packet loss) somewhere in the route is always bad and a sign of problems
Also Check out our VPS Hosting Page