Understanding Traceroute (Part 4 of 4)

A uni-directional traceroute (run only in one direction) can be used to accurately spot a problem area in a route.

If one does see some issues in a traceroute – that start at one hop and carry on forward (e.g. packet loss, or latency past the expected given the distance traveled between the two hops3), then it is imperative to run it ­bi-directionally. What I mean by this is, run, if possible at the same time or as close together time-wise as possible, a route from the user host to the server, and from the server to the user host.

It is important to do this, as often routes are asymmetric (take one path in one direction, yet another path in the return direction), and one needs to see the route both ways to be able to more accurately pinpoint where a problem may lie.

For example, trace from a customer to their server (this example now also brings together all the matters discussed in this article):

Tracing route to cyberfret.com [209.103.172.27] over a maximum of 30 hops:

1 7 ms 7 ms 9 ms 10.33.96.1
2 7 ms 9 ms 17 ms gig2-1.ghnaoh1-ybr1.columbus.rr.com [24.95.81.225] 3 15 ms 9 ms 9 ms srp9-0.clmboh1-rtr2.columbus.rr.com [65.25.129.94] 4 22 ms 20 ms 20 ms pos14-0.clmboh1-rtr3.columbus.rr.com [65.25.128.138] 5 22 ms 20 ms 19 ms son0-1-2.mtgmoh1-rtr0.columbus.rr.com [65.25.128.229] 6 25 ms 24 ms 22 ms te-3-2.car1.Cincinnati1.Level3.net [4.78.216.13] 7 23 ms 31 ms 34 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.132.206] 8 22 ms 35 ms 35 ms ae-78.ebr3.Chicago1.Level3.net [4.69.134.62] 9 31 ms 33 ms 35 ms ae-2.ebr2.Washington1.Level3.net [4.69.132.70] 10 30 ms 31 ms 54 ms ae-82-82.csw3.Washington1.Level3.net [4.69.134.154] 11 * * * Request timed out.
12 38 ms 38 ms 40 ms ge6-2.core1.dca2.hopone.net [66.36.224.190] 13 * * * Request timed out.
14 34 ms 34 ms 32 ms client.covesoft.net [209.103.172.27]

And a trace from their server back to their user host:

traceroute to 24.95.81.225 (24.95.81.225), 64 hops max, 40 byte packets
1 c-vl101-d1.acc.dca2.hopone.net (66.36.230.2) 0.381 ms 0.322 ms 0.241 ms
2 gec2.core2.dca2.hopone.net (66.36.224.233) 0.213 ms 0.199 ms 0.167 ms
3 gi1-0-0.ar2.wdc2.gblx.net (66.36.224.169) 1.339 ms 1.406 ms 1.368 ms
4 te1-1-10g.ar3.DCA3.gblx.net (67.17.108.134) 2.887 ms 2.585 ms 2.523 ms
5 pop2-ash-S2-1-0.atdn.net (66.185.136.9) 2.488 ms 2.608 ms 2.444 ms
6 bb2-ash-P3-0.atdn.net (66.185.151.150) 1.878 ms 1.941 ms 1.894 ms
7 bb2-cin-P3-0.atdn.net (66.185.153.61) 22.294 ms 22.182 ms 22.302 ms
8 pop1-cin-P1-0.atdn.net (66.185.133.3) 22.655 ms 22.407 ms 22.752 ms
9 RR-Cincinnati.atdn.net (66.185.133.6) 23.632 ms 23.698 ms 23.783 ms
10 pos13-1.clmboh1-rtr2.columbus.rr.com (24.95.81.194) 27.110 ms 26.940 ms 26.829 ms
11 srp1-1.ghnaoh1-ybr1.columbus.rr.com (65.25.129.68) 27.991 ms * 28.186 ms

As clearly evident, the route here is asymmetrical (as is often the case, given the various different routing preferences among various providers (and multiple, often countless, routes from point A to point B), each selecting the “best” routes differently from others – many often do that based on cost vs. performance, though the hopone.net network we use always does everything 100% based on optimal performance: shortest path, lowest latency, and always no packet loss). The route in (from customer to their server) goes from Road Runner in Columbus, OH to Level 3 in Cincinnati, OH and from there to HopOne in McLean, VA (the Level 3 [Old Meadow Road] McLean, VA site is misleadingly called Washington.Level3.net). The route out (from server to customer host), however, goes from HopOne in McLean, VA to GlobalCrossing in Washington, DC, then to AOL in Ashburn, VA and then to Road Runner in Cincinnati, OH, from where it travels to the end host in Columbus, OH.

The latency on the out route is lower, as it is tracing to the first real (not 10.x.x.x or 192.x.x.x non-routable IPs, that are not routable, private IP space used by various networks) hop in the customer’s in route, since the customer did not provide us with their actual host IP. As is evident in the in route, the customer has 7ms latency from their cable modem to their first Internet hop. As the out route traces to their gateway, it is, as expected, 7ms lower, as it does not trace all the way to the customer’s end host that is 7ms further away.

The in route shows two hops as “* * * Request timed out.” that, as earlier explained in this article, are no reason for concern, as long as the end host is reached with no problems. One of the hops, hop 11, is a Level 3 router that has been configured not to respond to traceroutes or pings (most likely for security and/or performance reasons), while the other hop, hop 13, is a HopOne distribution switch that is in non-routable IP space – which typically will show up in normal traceroutes (as hop 12 does), but in this case does not. However, as the end host is reached and shows no packet loss (3/3 pings to it have arrived perfectly fine, and all with consistent latency), it is evident that no problems are present. To ensure, however, the customer in this case would be suggested to run a recursive ping (ping –t) a 100+ times from their end host to the server, to verify that there are indeed no issues (there should not be, as the routes both ways are as close to perfect as possible, with stable latencies and no real packet loss anywhere). If there are any issues, such would be most likely present on the Road Runner end in Columbus, OH, as in the outgoing trace there are some packets lost there at the customer’s end gateway hop (however 1/3 packet loss does not have enough samples to go by to be objective, and it most likely is not reflective of 33% packet loss on that end host – as it is a router, it likely treats ping with the lowest priority and has a maximum of ping requests responses per second configured, so while it may be an indication of an issue there, it may be also simply the router doing what it has been configured to do and reject frequent pings).

/mtr –report-cycles=30 –report extby.neolocation.net
HOST: salviori.us.ded.neolocation Loss% Snt Last Avg Best Wrst StDev
1. h-vl202-d3.acc.dca2.hopone.n 0.0% 30 0.4 0.6 0.4 2.6 0.6
2. gec3.core2.dca2.hopone.net 0.0% 30 0.4 0.5 0.4 2.2 0.4
3. ge6-0.core1.dca2.hopone.net 0.0% 30 0.4 0.5 0.3 3.5 0.6
4. cw.peer.nyc2.hopone.net 0.0% 30 5.9 11.2 5.6 117.3 21.3
5. so-3-0-0-dcr1.nyk.cw.net 0.0% 30 5.8 13.0 5.7 121.9 27.2
6. so-7-0-0-dcr1.was.cw.net 0.0% 30 6.2 11.7 6.1 98.9 19.0
7. so-0-0-0-dcr1.par.cw.net 0.0% 30 86.4 91.2 86.3 136.4 11.7
8. so-7-0-0-dcr1.fra.cw.net 0.0% 30 103.4 97.0 94.9 124.1 5.5
9. so-4-0-0-zar1.fri.cw.net 0.0% 30 95.1 96.1 94.9 120.0 4.5
10. romteleco-gw.fri.cw.net 50.0% 30 455.1 462.5 450.4 532.3 23.3
11. spb-dsr2-ge0-2-0-0.rt-comm.r 40.0% 30 488.0 492.3 484.1 568.6 19.1
12. 195.161.4.34 26.7% 30 486.7 487.0 484.2 491.4 1.7
13. J10-1-B57.ge-1-0-0-7.peterst 56.7% 30 488.1 487.4 485.3 488.8 1.2
14. BELPAK.peterstar.net 60.0% 30 498.7 498.3 494.5 501.6 1.9
15. 193.232.248.110 56.7% 30 497.8 519.5 494.1 778.8 77.9
16. 193.232.249.185 53.3% 30 499.9 499.2 494.5 503.3 2.7
17. 86.57.250.98 63.3% 30 498.0 500.6 495.3 521.4 7.4

In this route, it is evident in the highlighted section, that there is an issue with “romteleco” (rt-comm.ru) gateway to their IP transit provider, Cable & Wireless (cw.net) – it would seem that this their gateway to C&W is seriously overloaded/congested, as the high packet loss carries all the way through past it on every hop. At such high packet loss (27-60%), the connectivity is likely to be severely limited.

In this particular case, an incoming route (from the customer to their server) would likely show the opposite: packet loss always after the “romteleco” gateway with C&W and onwards from there, as that is where the issue is caused. Seeing such an incoming route to compare it with the outgoing route would then confirm where the problem lies, if in both direction of routes it starts happening after one particular hop (in this case, the “romteleco” link to C&W).

Understanding Traceroute: A Four-part Series

Part 1: Traceroute can be used to measure performance

Part 2: Traceroute/tracert-like applications or commands will provide accurate results

Part 3: * * * s (apparent packet loss) somewhere in the route is always bad and a sign of problems

Part 4: A uni-directional traceroute (run only in one direction) can be used to accurately spot a problem area in a route

Also Check out our VPS Hosting Page

Happy Birthdays

Google Birthday logosThe online world was abuzz yesterday with the official 9th birthday of search engine giant Google – I’m sure you’ve heard of it? Although co-founders Larry Page and Sergey Brin registered the domain in 1997, the site was officially launched a year later. As usual, Google’s trademark logo was adapted for the event. For an overview of Google’s history, see this link.

In other birthday news, Dreamhost celebrated their 10th birthday on September 24th. Another one of the old boys in the hosting industry, there is a great post on the Dreamhost blog chronicling their evolution over the last 10 years. Although I’m not sure it competes with Google’s birthday logos, it’s not a bad (light) read. (For the record, I’ve already put my name in for the 12.5 PB of storage and $1.95/month bandwidth that will be offered in 2017.)

Congratulations to both these companies on these landmarks that would make them ancient in terms of online/Tech companies – but neither as old as Superb!

UPDATE: I almost forgot – Superb had some birthdays that are definitely important enough to mention alongside Google and Dreamhost; Happy Birthday to Andree, David, and Lindsay. 🙂

Web-based Applications

One of the causes of the dot-com bust was the inability of companies to deliver the rich web-based applications that users expected, and although the limitations on the development of these apps was tied directly to the technology available to the average end-user attempting to access them (hardly the fault of the developers or those trying to deliver the apps), the frustrations experienced by customers was unavoidable and prevented widespread adoption. That’s all changed.

Flash was an interesting, early attempt to deliver a next-level user experience, but the attempts to use Flash effectively are quite hit-and-miss, and the major updates to the platform have been in the multimedia arena and has done little for web apps. Ajax was a huge step for both developers and users, and many of the best and most useful web-based applications take advantage of the web development technique.

An article released yesterday on Computerworld provides a handful of free web-based applications the they can’t live without.

Five free Web apps we can’t live without

From collaboration tools to database apps and more, these next-gen Web applications keep the Computerworld newsroom humming.

Web apps we can’t live without:

The A-List

Honourable Mentions

Search Engine Rankings

One of the most common questions people ask regarding their website is how to get it ranked well, usually not on search engines, but more specifically, on Google. While rankings on the major search engines are not universal, there is quite a bit of parity among them, and a site that we sponsor, SEOMoz, is one of the best in the business at providing tips and tricks for ranking well on search engines. In an article written earlier this week, Rand Fishkin outlines (in his opinion) the 4 most important variables in the search engine ranking equation:

  1. #1 – Keyword Usage & Content Relevance
  2. #2 – Raw Link Juice
  3. #3 – Anchor Text Weight
  4. #4 – Domain Authority

Click here for the full article.

It is important to note that search engine rankings are not a perfect science, and there are so many different things that can affect your site’s ranking(s); please review the SEOMoz Blog Disclaimer. While it is important to rank well on search engines, it is possible to have a successful Internet business and not rank well on search engines through word-of-mouth by existing customers and/or a well executed, targeted marketing plan.

Silverlight for Web Hosting

Earlier this month, I posted about Microsoft Silverlight, comparing the recent release from the software giant to Apple’s iPhone. While the adoption of Microsoft’s new web platform may not rival that of the iPhone, which recently reached the 1 million units sold milestone, websites, seminars, and tutorials will slowly become available. Michael Sherotter, an evangelist for Microsoft’s Communications Sector, […] tasked with explaining Microsoft’s latest technologies, announced a Webcast earlier this month on Silverlight for Web Hosting.

Silverlight, Microsoft’s new cross-platform web browser technology for media playback and rich Internet applications (RIAs) represents a great opportunity for Web Hosting Companies to differentiate themselves and provider new services to their customers.

  • What: Microsoft Silverlight For Web Hosting Companies
  • When: Friday, September 28th, 2007, 11:00 AM to 12:30 PM Pacific Time
  • Where: Microsoft LiveMeeting over the Internet
  • Why: Demand is growing for media hosting & streaming and rich interactive applications on the Internet and web hosting companies need to understand the implications for them.

The session will be recorded, and should be a great way to get acquainted with Silverlight.

Network & Server Security

Network/Server SecuritySecurity relating to computers and networks has always been a concern for IT managers tending to Enterprise-class operations. Despite all their efforts to keep their networks free from intruders – be it a hacker, a worm, a trojan, or a virus – the biggest security risk to these systems is most often the users themselves. Over time, more and more businesses have started to depend on Technology and their hardware infrastructure, for their daily operations, and as these aspects of a business have become more critical, these hackers, worms, and trojans have become more targetted, again, typically focussing on the users. Instead of coming from the 13-year-old computer aficianado looking for some fun and fame, organized teams have been setup with a specific target, which, more often than not, is data.

In the last year, security flaws or breaches at large corporations have resulted in individuals being at-risk. Consumer information, from search results to personal credit card and debit card numbers, has been compromised; sometimes it’s a simple (albeit costly) mistake from an individual, but with multi-national corporations and millions of dollars potentially at stake, insider breaches are also a concern. But what does this mean for the average business?

All companies with sensitive data need to be aware that they are a target. Unless the proper steps are taken to ensure that your networks are secure, your data and your systems will be susceptible to attacks. The average webmaster or designated ‘IT guy’ in the company will not have the ability to maintain this level of security, and these types of services may require an outside resource to perform security audits on your systems. It’s also important to note the differences between network and server security.

For those that take advantage of co-location or a hosted server from companies like Superb, network security isn’t the issue; instead, keeping up-to-date with patches and updates for the server’s operating system and maintaining solid, secure coding practices is key to preventing unauthorized access. To help prevent unnecessary risk, we (the Superb Team) are putting together an unofficial checklist for self-managed servers, but it is definitely recommended that a professional review your server security regularly.