Technical (internet) help PLEASE...

[mmblz]

Hi all, we have two admins, as well as a handful of users, who cannot access the site. The connection times out. Doing a traceroute (whether to our domain name, or directly to ip address) gets stuck at a
certain address.
In cases where everything is working, the traceroute shows that the address where other people gets stuck is the one hop right before our server.
68Trishield and floydp both started off being able to get to the site but then suddenly could not, and have been stuck not being able to for over 24 hours.
I briefly got stuck out for about an hour, but then it started working again.
On a different computer I can get remote to (in NH), it was not working. Then 12 hours or so later it was. I did a ping and everything was fine. Then I tried a traceroute and it started to time out. Then I tried to ping again and it failed. This was all to IP, so DNS is not an issue...
Very very strange. If anyone has any ideas, please post or PM me...

:confused:

[Neuromancer]

Where, physically, is the host server?

[markem]

Where, physically, is the host server?

Just east of LA

[smitdavi]

If it makes a difference I'm with AT&T U-Verse and using a 2wire

[Neuromancer]

Just east of LA
Well, LA area should provide plenty of coverage connection-wise...any idea who, what, or where the hop right before the CA server is? And what ISP's are you guys using, that are having trouble?

[smitdavi]

Well, LA area should provide plenty of coverage connection-wise...any idea who, what, or where the hop right before the CA server is? And what ISP's are you guys using, that are having trouble?

I'm having the trouble....pm me if you guys want the blocked IP#

[markem]

Well, LA area should provide plenty of coverage connection-wise...any idea who, what, or where the hop right before the CA server is? And what ISP's are you guys using, that are having trouble?

I've asked for copies of the traceroutes. It could be a failed/failing router, lack of convergence for paths (OSPF/BGP or something else - MPLS tagging label corruption) or something else. No real idea until I see some results.

[smitdavi]

I've asked for copies of the traceroutes. It could be a failed/failing router, lack of convergence for paths (OSPF/BGP or something else - MPLS tagging label corruption) or something else. No real idea until I see some results.

I didn't understand a word you said lol. Good luck though :tu

[Dood]

OK, this is wierd.
Here is the BAD traceroute:
Traceroute has started ...
traceroute to www.cigarasylum.com (http://www.cigarasylum.com) (67.222.135.200), 64 hops max, 40 byte packets
1 home (192.168.1.254) 24.656 ms 1.098 ms 1.002 ms
2 99-149-24-2.lightspeed.bltnin.sbcglobal.net (99.149.24.2) 28.509 ms 29.287 ms 25.818 ms
3 * * *
4 * * *
5 * * 75.19.192.2 (75.19.192.2) 36.797 ms
6 151.164.41.194 (151.164.41.194) 27.207 ms 25.328 ms 34.198 ms
7 151.164.94.41 (151.164.94.41) 41.183 ms 38.860 ms 38.992 ms
8 te-3-2.chicago1.level3.net (4.68.110.197) 40.368 ms 44.892 ms 39.508 ms
9 ae-32-56.ebr2.chicago1.level3.net (4.68.101.190) 48.008 ms 45.823 ms 56.251 ms
10 ae-1-100.ebr1.chicago1.level3.net (4.69.132.41) 58.352 ms ae-5.ebr2.chicago2.level3.net (4.69.140.194) 53.703 ms 39.463 ms
11 ae-1-100.ebr1.chicago2.level3.net (4.69.132.113) 38.291 ms 38.511 ms 39.360 ms
12 ae-3.ebr2.denver1.level3.net (4.69.132.61) 77.144 ms 66.908 ms 86.074 ms
13 ae-1-100.ebr1.denver1.level3.net (4.69.132.37) 66.789 ms 70.088 ms 79.139 ms
14 ae-2.ebr2.dallas1.level3.net (4.69.132.106) 73.161 ms 78.013 ms 66.266 ms
15 ae-72-72.csw2.dallas1.level3.net (4.69.136.142) 70.387 ms 78.529 ms ae-82-82.csw3.dallas1.level3.net (4.69.136.146) 77.760 ms
16 ae-31-89.car1.dallas1.level3.net (4.68.19.131) 65.803 ms 64.888 ms 92.412 ms
17 colo4-dalla.car1.dallas1.level3.net (8.9.232.74) 68.006 ms 66.169 ms 69.834 ms
18 72.249.0.66 (72.249.0.66) 69.150 ms 66.539 ms 67.520 ms
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * *

IP Address: 192.168.1.76
Here is a GOOD traceroute(from my PC):

Here is my good traceroute:
Tracing route to www.cigarasylum.com (http://www.cigarasylum.com) [67.222.135.200]
over a maximum of 30 hops:
1 2 ms 3 ms 1 ms 192.168.1.1
2 5 ms 2 ms 3 ms 10.0.0.1
3 63 ms 59 ms 59 ms clsp-dsl-gw05-197.clsp.qwest.net [67.42.184.197]
4 60 ms 60 ms 59 ms clsp-agw1.inet.qwest.net [67.42.184.125]
5 61 ms 59 ms 59 ms cls-core-01.inet.qwest.net [205.171.152.65]
6 83 ms 82 ms 83 ms 67.14.2.85
7 83 ms 94 ms 82 ms xe-8-2-0.edge2.dallas3.level3.net [4.68.63.53]
8 190 ms 107 ms 199 ms ae-11-69.car1.Dallas1.Level3.net [4.68.19.3]
9 87 ms 85 ms 87 ms COLO4-DALLA.car1.Dallas1.Level3.net [8.9.232.74]
10 84 ms 84 ms 84 ms 72.249.0.66
11 84 ms 84 ms 98 ms www.cigarasylum.com (http://www.cigarasylum.com) [67.222.135.200]
Trace complete.
72.249.0.66 is the last router before the website.
Don't know why it isn't passing some requests.

[markem]

OK, this is wierd.

72.249.0.66 is the last router before the website.
Don't know why it isn't passing some requests.


Here is the owner information for that router. The contact info is
Phone: +1-214-630-3100 27 (Office)
Email: support@colo4dallas.com

NetRange: 72.249.0.0 (http://ws.arin.net/whois/?queryinput=72.249.0.0) - 72.249.191.255 (http://ws.arin.net/whois/?queryinput=72.249.191.255)
CIDR: 72.249.0.0/17, 72.249.128.0/18
NetName: COLO4-BLK2 (http://ws.arin.net/whois/?queryinput=N%20.%20COLO4-BLK2)
NetHandle: NET-72-249-0-0-1 (http://ws.arin.net/whois/?queryinput=N%20%21%20NET-72-249-0-0-1)
Parent: NET-72-0-0-0-0 (http://ws.arin.net/whois/?queryinput=N%20NET-72-0-0-0-0)
NetType: Direct Allocation
NameServer: NS1.COLO4DALLAS.NET
NameServer: NS2.COLO4DALLAS.NET
Comment:
RegDate: 2006-08-25
Updated: 2007-10-17

OrgAbuseHandle: CAM9-ARIN (http://ws.arin.net/whois/?queryinput=P%20%21%20CAM9-ARIN)
OrgAbuseName: Colo4Dallas Abuse Manager
OrgAbusePhone: +1-214-630-3100
OrgAbuseEmail: abuse@colo4dallas.com

OrgNOCHandle: NOC1718-ARIN (http://ws.arin.net/whois/?queryinput=P%20%21%20NOC1718-ARIN)
OrgNOCName: NOC
OrgNOCPhone: +1-214-630-3100
OrgNOCEmail: support@colo4dallas.com

OrgTechHandle: CAR47-ARIN (http://ws.arin.net/whois/?queryinput=P%20%21%20CAR47-ARIN)
OrgTechName: Colo4Dallas ARIN Requests
OrgTechPhone: +1-214-630-3100

OrgTechEmail: arin@colo4dallas.com

The network operations center may be able to help
Name: NOC
Handle: NOC1718-ARIN (http://ws.arin.net/whois/?queryinput=NOC1718-ARIN)
Company: Colo4Dallas
Address: 3000 Irving Blvd
City: Dallas
StateProv: TX
PostalCode: 75247
Country: US
Comment:
RegDate: 2005-01-25
Updated: 2007-02-12
Phone: +1-214-630-3100 27 (Office)
Email: support@colo4dallas.com

[smitdavi]

Thanks Mark...I'd owe ya big if we can figure this out.

[markem]

Do any of the admins know if the server has more than on network interface? They would all have the same IP address, since only only one is assigned to the domain name.

Also, if you call the NOC (network operations center), have them check their DNS to see if there is more than one IP address associated with the name internal to them. Also have them check the ARP cache to make sure that an invalid entry isn't stuck.

[smitdavi]

Do any of the admins know if the server has more than on network interface? They would all have the same IP address, since only only one is assigned to the domain name.

Also, if you call the NOC (network operations center), have them check their DNS to see if there is more than one IP address associated with the name internal to them. Also have them check the ARP cache to make sure that an invalid entry isn't stuck.

Mark do you need anything from me?

[kwoody]

Colo4dallas is a good company. (have used them myself)
Try emailing them and ask them why the routing errors, maybe on their end (since the hops do get to dallas)

[Hardcz]

Best thing you can do is contact the internet provider, if you're having problems so they can investigate. The way things should work is your connection goes to say comcast, and they connect to ATT/Verizon via a frame relay cloud, which picks the route to the CA server, wherever it is, and if it's not routing correctly, they can modify the route or find out if there's a problem. Most likely there's an outage somewhere that's causing the page to be unavailable. Something you could try is using an anonymous proxy server in your internet browser, which basically routes your traffic from their location to here, which assuming it's not on the same nodes, will find a route that's not bad.

[smitdavi]

Best thing you can do is contact the internet provider, if you're having problems so they can investigate. The way things should work is your connection goes to say comcast, and they connect to ATT/Verizon via a frame relay cloud, which picks the route to the CA server, wherever it is, and if it's not routing correctly, they can modify the route or find out if there's a problem. Most likely there's an outage somewhere that's causing the page to be unavailable. Something you could try is using an anonymous proxy server in your internet browser, which basically routes your traffic from their location to here, which assuming it's not on the same nodes, will find a route that's not bad.

So is that something that I need to do. Call AT&T?

[markem]

Mark do you need anything from me?

Not at this time, David. The problem looks to be very local. It is probably inside the co-lo facility where the server is housed.

[markem]

So is that something that I need to do. Call AT&T?

No, because it isn't your problem, it is CAs problem. Someone from CA would need to contact them or their account rep.

[smitdavi]

Not at this time, David. The problem looks to be very local. It is probably inside the co-lo facility where the server is housed.

Alright...let me know if you guys need any info or need for me to do anything on my end!

[mmblz]

Do any of the admins know if the server has more than on network interface? They would all have the same IP address, since only only one is assigned to the domain name.

yes, it has a number of network interfaces, with different ip addresses (afaik)

i've told them about the problem and they couldn't find any problem with configuration.

[Hardcz]

Not at this time, David. The problem looks to be very local. It is probably inside the co-lo facility where the server is housed.

Most Data centers will have two pipes or more, for redundancy, say if they're in Texas, having one go to Chicago and the other to LA or NY.... So as possible as it is, because a large amount of people still can get to it, I'd still think to look at the ISP end first. Users should contact their ISP only if having problems because if not, they'll show it as being fine. Now the admins should alert the web host as well to a problem, so they can investigate, but something like this unless you're on the inside isn't as easy to track down.

Basing this off the experience I have with working at an ISP previously and with out the networks my company and it's customers use, which are scattered across the US, and several other countries...having some locations store different servers for Apps and such.

[markem]

yes, it has a number of network interfaces, with different ip addresses (afaik)

i've told them about the problem and they couldn't find any problem with configuration.

You really want to talk to someone who can log on to the router right before the server and do some testing. That seems to be where the problem is at.

[Hardcz]

My BGP table is bigger than your BGP table

[markem]

Most Data centers will have two pipes or more, for redundancy, say if they're in Texas, having one go to Chicago and the other to LA or NY.... So as possible as it is, because a large amount of people still can get to it, I'd still think to look at the ISP end first. Users should contact their ISP only if having problems because if not, they'll show it as being fine. Now the admins should alert the web host as well to a problem, so they can investigate, but something like this unless you're on the inside isn't as easy to track down.

Basing this off the experience I have with working at an ISP previously and with out the networks my company and it's customers use, which are scattered across the US, and several other countries...having some locations store different servers for Apps and such.

Agreed, more or less. Since the problem is manifesting itself at the 1st upstream router, the problem, statistically, is local to the facility. That is, unless they do physical layer shunting across the US, which is very costly and make little or no sense.

The problem could be a myriad of things from a bad network interface on the server to a HW or SW issue on the upstream device (router or switch) or internal DNS configuration issue or similar.

There is always the possibility that the server is running IP chains or something similar and that the config is screwed up in some way, but that's way far fetched base on what I know.

Like you, I have load of network engineering experience plus telecommunications OS design, work on networking standards committees, and a prior life as a network programmer and college professor. That doesn't mean that either of us knows squat about this problem, however.

[Hardcz]

Like you, I have load of network engineering experience plus telecommunications OS design, work on networking standards committees, and a prior life as a network programmer and college professor. That doesn't mean that either of us knows squat about this problem, however.

This could be the start of a networking banter thread :ss

[markem]

This could be the start of a networking banter thread :ss

Duuuuude! :D

[poker]

Holy **** you guys lost me at hello :confused: LMAO

[Dood]

I would contribute a few packets!

[Hardcz]

mmblz, hope some of these thoughts end up helping out with the problems that are going on. Keep us updated and we'll keep throwing random crap your way.

[markem]

Julian and I talked on the phone. We think that the problem is isolated to the configuration to iptables on the server. iptables is a security program (more or less) and it seems to be logging blocked IPs that are corresponding to at least some of those having problems. He's calling the company that owns the server now.

[Sauer Grapes]

I know this doesn't help with the solution, but it's interesting that both admins that can't log on come from the mid atlantic area and are about two hours apart.

[mmblz]

couldn't reach the host yet.
i think the problem might have something to with csf (a program i hadn't heard of before) ;)

[markem]

couldn't reach the host yet.
i think the problem might have something to with csf (a program i hadn't heard of before) ;)

Interesting

http://www.configserver.com/cp/csf.html

[mmblz]

yeah i found a readme, etc
frank and dave got blocked for typing a bad htpasswd 5 times (lame), and i found the command to whitelist them
now have to figure out why random users would be blocked, or why traceroute would trigger blocking

[markem]

yeah i found a readme, etc
frank and dave got blocked for typing a bad htpasswd 5 times (lame), and i found the command to whitelist them
now have to figure out why random users would be blocked, or why traceroute would trigger blocking


ssswwwweeeeeeettttttt!

Way to go, Julian.

[Sauer Grapes]

yeah i found a readme, etc
frank and dave got blocked for typing a bad htpasswd 5 times (lame), and i found the command to whitelist them
now have to figure out why random users would be blocked, or why traceroute would trigger blocking
Must have been a senior moment for them. :ss

[Waynegro1]

:confused::confused:

What language is this?

[markem]

:confused::confused:

What language is this?

It's a language the some crazy people in an asylum made up to pass the time. ;)

[mmblz]

Alright...let me know if you guys need any info or need for me to do anything on my end!

see if it works next time you have a chance...

[smitdavi]

see if it works next time you have a chance...

no dice


yeah i found a readme, etc
frank and dave got blocked for typing a bad htpasswd 5 times (lame), and i found the command to whitelist them
now have to figure out why random users would be blocked, or why traceroute would trigger blocking

julian....when the site first went live it kept prompting me for a password. I just hit continue or entered a random password. It always rejected it....could I have the same problem as Frank and Dave?

[mmblz]

no dice




julian....when the site first went live it kept prompting me for a password. I just hit continue or entered a random password. It always rejected it....could I have the same problem as Frank and Dave?

i suppose it is vaguely possible...
though i did search for your ip, etc.. but..

i just flushed out the couple of blocked ips that i didn't think were you but maybe were.
try once more :D

[Hardcz]

Julian....think we'll need to make you an honorary Network Admin when this is done.

[smitdavi]

i suppose it is vaguely possible...
though i did search for your ip, etc.. but..

i just flushed out the couple of blocked ips that i didn't think were you but maybe were.
try once more :D

Wohoo...it works. Whatever switch you flipped was apparently the right one. Thanks :tu

[King James]

problem solved!

[mmblz]

Wohoo...it works. Whatever switch you flipped was apparently the right one. Thanks :tu

thank GOD :tu

[smitdavi]

thank GOD :tu

sorry to be a pain. I know you guys have been balls to the wall for a while now. This has turned into one hell of a place in just a short while. You guys should be proud. I'm def grateful :tu

[mmblz]

no pain at all - it was the overly protective csf settings that were a pain ;)