Networking

I’ve been taking a look at our resolvers and I was surprised by some of the results I found.
I ran a tcpdump for 10 minutes capturing packets sent to one of our resolvers and extracted the names being queried.
During those 10 minutes that particular resolver answered 1,25 million queries for 250 thousand distinct names.
Looking through the list there were many names that result from mis-configured equipment and other mistakes, but that’s on the low end of queries. It’s the higher end, with the most commonly resolved names that actually interests us.
The list is topped by a name that is hard-coded into some of our clients routers. Having several hundred thousand of those devices out in the open making queries does skew the results so I ignored those queries, and just looked at the rest of the names. I ordered them by frequency and here is a brief analysis of the top 50 names.

As one might expect, at the top of the list comes ‘www.facebook.com’ but I was actually surprised to find so many names related to facebook. There are also ‘static.ak.fbcdn.net’, ‘apps.facebook.com’, ‘profile.ak.fbcdn.net’, ‘pixel.facebook.com’, ‘creative.ak.fbcdn.net’, ‘platform.ak.fbcdn.net’, ‘external.ak.fbcdn.net’, ‘static.ak.connect.facebook.com’, ‘photos-g.ak.fbcdn.net’, ‘photos-b.ak.fbcdn.net’, ‘photos-e.ak.fbcdn.net’, ‘static.ak.facebook.com’, ‘photos-c.ak.fbcdn.net’, ‘photos-a.ak.fbcdn.net’, and if I had dug deeper, I would certainly have found more names.
In case you haven’t figured it out fbcdn stands for facebook content delivery network, and ak means Akamai.
Out of the top 50 names queried, 15 belong or are related to Facebook. That is impressive.

The second most popular name being queried was a root server. Not sure I understand why, but there were many, many queries resolving ‘a.root-servers.net’. A close third was Google’s ‘www.google-analytics.com’. No surprise here, as it is probably the most widely used analytics solution today.
Fourth place was used by our own voip proxy, which is always nice to see 🙂
In fifth place we have ‘google.com’, followed by ‘www.youtube.com’, and ‘www.google.com’. Funny that our local ‘www.google.pt’ only made 13th place.
Also related to youtube are some names like ‘i1.ytimg.com’, ‘i2.ytimg.com’, ‘i3.ytimg.com’, ‘i4.ytimg.com’ that show up at the lower end of the 50.
There is also ‘googleads.g.doubleclick.net’, and ‘pagead2.googlesyndication.com’ which are self-explanatory.

Then there are a couple of ntp servers, and at least 1 anti-virus name I recognize.

This was just a trial run, and I found the results pretty interesting.
Maybe I can automate this, and see what other surprises hide lurking in the data.

One of the things I have on my (short) list to blog about is SNMP.

I’ve been thinking about how to approach it, and today, one of the sysadmins I follow on twitter (@standaloneSA) tweeted that he had written an entry about SNMP. I went to his blog to check it out, and I highly recommend it. It’s much better than what I would have written, so I’ll just point people his way.

You can reach his blog entry here.  Well done, Matt.

This is something I found online, on a linux site I have since forgotten the name. (Ping me if you know it’s origin, so I can give proper credit to the author)
Anyway, I have it in my shell’s alias’ file:

nn='netstat -an | grep ESTABLISHED | awk '\''{print $5}'\'' | awk -F: '\''{print $1}'\'' | sort | uniq -c | awk '\''{ printf("%s\t%s\t",$2,$1); for (i = 0; i < $1; i++) {printf("*")}; print ""}'\'''

It’s rather useful, as I can quickly see a graphic representation of my TCP connections, in the ESTABLISHED state.
The output is something like this:

~$ nn
10.XXX.XX.XX    1       *
10.XXX.XX.XX    5       *****
10.XXX.XX.X     2       **
143.XXX.XXX.XX  1       *
192.XXX.X.XX    1       *
208.XX.XXX.XX   1       *
212.XXX.XXX.XX  1       *
213.XX.XXX.XX   1       *

For security reasons I mangled the IP addresses with X’s. This was taken on my work PC, but imagine running it on a webserver. It might just help you figure out who’s sucking up your connections (or not).
I recently had a problem with a webserver, and this little snippet led me to find out that a certain IP address had 250 established connections to each of the frontend servers of that particular service. One iptables command later, and we could breathe again…