Technical (internet) help PLEASE...
 

Hi all, we have two admins, as well as a handful of users, who cannot access the site. The connection times out. Doing a traceroute (whether to our domain name, or directly to ip address) gets stuck at a
certain address.
In cases where everything is working, the traceroute shows that the address where other people gets stuck is the one hop right before our server.
68Trishield and floydp both started off being able to get to the site but then suddenly could not, and have been stuck not being able to for over 24 hours.
I briefly got stuck out for about an hour, but then it started working again.
On a different computer I can get remote to (in NH), it was not working. Then 12 hours or so later it was. I did a ping and everything was fine. Then I tried a traceroute and it started to time out. Then I tried to ping again and it failed. This was all to IP, so DNS is not an issue...
Very very strange. If anyone has any ideas, please post or PM me...

:confused:

Re: Technical (internet) help PLEASE...
 

Where, physically, is the host server?

Re: Technical (internet) help PLEASE...
 

Just east of LA

Re: Technical (internet) help PLEASE...
 

If it makes a difference I'm with AT&T U-Verse and using a 2wire

Re: Technical (internet) help PLEASE...
 

Well, LA area should provide plenty of coverage connection-wise...any idea who, what, or where the hop right before the CA server is? And what ISP's are you guys using, that are having trouble?

Re: Technical (internet) help PLEASE...
 

I'm having the trouble....pm me if you guys want the blocked IP#

Re: Technical (internet) help PLEASE...
 

I've asked for copies of the traceroutes. It could be a failed/failing router, lack of convergence for paths (OSPF/BGP or something else - MPLS tagging label corruption) or something else. No real idea until I see some results.

Re: Technical (internet) help PLEASE...
 

I didn't understand a word you said lol. Good luck though :tu

Re: Technical (internet) help PLEASE...
 

OK, this is wierd.
Here is the BAD traceroute:
Traceroute has started ...
traceroute to www.cigarasylum.com (67.222.135.200), 64 hops max, 40 byte packets
1 home (192.168.1.254) 24.656 ms 1.098 ms 1.002 ms
2 99-149-24-2.lightspeed.bltnin.sbcglobal.net (99.149.24.2) 28.509 ms 29.287 ms 25.818 ms
3 * * *
4 * * *
5 * * 75.19.192.2 (75.19.192.2) 36.797 ms
6 151.164.41.194 (151.164.41.194) 27.207 ms 25.328 ms 34.198 ms
7 151.164.94.41 (151.164.94.41) 41.183 ms 38.860 ms 38.992 ms
8 te-3-2.chicago1.level3.net (4.68.110.197) 40.368 ms 44.892 ms 39.508 ms
9 ae-32-56.ebr2.chicago1.level3.net (4.68.101.190) 48.008 ms 45.823 ms 56.251 ms
10 ae-1-100.ebr1.chicago1.level3.net (4.69.132.41) 58.352 ms ae-5.ebr2.chicago2.level3.net (4.69.140.194) 53.703 ms 39.463 ms
11 ae-1-100.ebr1.chicago2.level3.net (4.69.132.113) 38.291 ms 38.511 ms 39.360 ms
12 ae-3.ebr2.denver1.level3.net (4.69.132.61) 77.144 ms 66.908 ms 86.074 ms
13 ae-1-100.ebr1.denver1.level3.net (4.69.132.37) 66.789 ms 70.088 ms 79.139 ms
14 ae-2.ebr2.dallas1.level3.net (4.69.132.106) 73.161 ms 78.013 ms 66.266 ms
15 ae-72-72.csw2.dallas1.level3.net (4.69.136.142) 70.387 ms 78.529 ms ae-82-82.csw3.dallas1.level3.net (4.69.136.146) 77.760 ms
16 ae-31-89.car1.dallas1.level3.net (4.68.19.131) 65.803 ms 64.888 ms 92.412 ms
17 colo4-dalla.car1.dallas1.level3.net (8.9.232.74) 68.006 ms 66.169 ms 69.834 ms
18 72.249.0.66 (72.249.0.66) 69.150 ms 66.539 ms 67.520 ms
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * *

IP Address: 192.168.1.76
Here is a GOOD traceroute(from my PC):

Here is my good traceroute:
Tracing route to www.cigarasylum.com [67.222.135.200]
over a maximum of 30 hops:
1 2 ms 3 ms 1 ms 192.168.1.1
2 5 ms 2 ms 3 ms 10.0.0.1
3 63 ms 59 ms 59 ms clsp-dsl-gw05-197.clsp.qwest.net [67.42.184.197]
4 60 ms 60 ms 59 ms clsp-agw1.inet.qwest.net [67.42.184.125]
5 61 ms 59 ms 59 ms cls-core-01.inet.qwest.net [205.171.152.65]
6 83 ms 82 ms 83 ms 67.14.2.85
7 83 ms 94 ms 82 ms xe-8-2-0.edge2.dallas3.level3.net [4.68.63.53]
8 190 ms 107 ms 199 ms ae-11-69.car1.Dallas1.Level3.net [4.68.19.3]
9 87 ms 85 ms 87 ms COLO4-DALLA.car1.Dallas1.Level3.net [8.9.232.74]
10 84 ms 84 ms 84 ms 72.249.0.66
11 84 ms 84 ms 98 ms www.cigarasylum.com [67.222.135.200]
Trace complete.
72.249.0.66 is the last router before the website.
Don't know why it isn't passing some requests.

Re: Technical (internet) help PLEASE...
 

Here is the owner information for that router. The contact info is
Phone: +1-214-630-3100 27 (Office)
Email: support@colo4dallas.com

NetRange: 72.249.0.0 - 72.249.191.255
CIDR: 72.249.0.0/17, 72.249.128.0/18
NetName: COLO4-BLK2
NetHandle: NET-72-249-0-0-1
Parent: NET-72-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.COLO4DALLAS.NET
NameServer: NS2.COLO4DALLAS.NET
Comment:
RegDate: 2006-08-25
Updated: 2007-10-17

OrgAbuseHandle: CAM9-ARIN
OrgAbuseName: Colo4Dallas Abuse Manager
OrgAbusePhone: +1-214-630-3100
OrgAbuseEmail: abuse@colo4dallas.com

OrgNOCHandle: NOC1718-ARIN
OrgNOCName: NOC
OrgNOCPhone: +1-214-630-3100
OrgNOCEmail: support@colo4dallas.com

OrgTechHandle: CAR47-ARIN
OrgTechName: Colo4Dallas ARIN Requests
OrgTechPhone: +1-214-630-3100

OrgTechEmail: arin@colo4dallas.com

The network operations center may be able to help
Name: NOC
Handle: NOC1718-ARIN
Company: Colo4Dallas
Address: 3000 Irving Blvd
City: Dallas
StateProv: TX
PostalCode: 75247
Country: US
Comment:
RegDate: 2005-01-25
Updated: 2007-02-12
Phone: +1-214-630-3100 27 (Office)
Email: support@colo4dallas.com

Re: Technical (internet) help PLEASE...
 

Thanks Mark...I'd owe ya big if we can figure this out.

Re: Technical (internet) help PLEASE...
 

Do any of the admins know if the server has more than on network interface? They would all have the same IP address, since only only one is assigned to the domain name.

Also, if you call the NOC (network operations center), have them check their DNS to see if there is more than one IP address associated with the name internal to them. Also have them check the ARP cache to make sure that an invalid entry isn't stuck.

Re: Technical (internet) help PLEASE...
 

Mark do you need anything from me?

Re: Technical (internet) help PLEASE...
 

Colo4dallas is a good company. (have used them myself)
Try emailing them and ask them why the routing errors, maybe on their end (since the hops do get to dallas)

Re: Technical (internet) help PLEASE...
 

Best thing you can do is contact the internet provider, if you're having problems so they can investigate. The way things should work is your connection goes to say comcast, and they connect to ATT/Verizon via a frame relay cloud, which picks the route to the CA server, wherever it is, and if it's not routing correctly, they can modify the route or find out if there's a problem. Most likely there's an outage somewhere that's causing the page to be unavailable. Something you could try is using an anonymous proxy server in your internet browser, which basically routes your traffic from their location to here, which assuming it's not on the same nodes, will find a route that's not bad.

Re: Technical (internet) help PLEASE...
 

So is that something that I need to do. Call AT&T?

Re: Technical (internet) help PLEASE...
 

Not at this time, David. The problem looks to be very local. It is probably inside the co-lo facility where the server is housed.

Re: Technical (internet) help PLEASE...
 

No, because it isn't your problem, it is CAs problem. Someone from CA would need to contact them or their account rep.

Re: Technical (internet) help PLEASE...
 

Alright...let me know if you guys need any info or need for me to do anything on my end!

Re: Technical (internet) help PLEASE...
 

yes, it has a number of network interfaces, with different ip addresses (afaik)

i've told them about the problem and they couldn't find any problem with configuration.

Re: Technical (internet) help PLEASE...
 

Most Data centers will have two pipes or more, for redundancy, say if they're in Texas, having one go to Chicago and the other to LA or NY.... So as possible as it is, because a large amount of people still can get to it, I'd still think to look at the ISP end first. Users should contact their ISP only if having problems because if not, they'll show it as being fine. Now the admins should alert the web host as well to a problem, so they can investigate, but something like this unless you're on the inside isn't as easy to track down.

Basing this off the experience I have with working at an ISP previously and with out the networks my company and it's customers use, which are scattered across the US, and several other countries...having some locations store different servers for Apps and such.

Re: Technical (internet) help PLEASE...
 

You really want to talk to someone who can log on to the router right before the server and do some testing. That seems to be where the problem is at.

Re: Technical (internet) help PLEASE...
 

My BGP table is bigger than your BGP table

Re: Technical (internet) help PLEASE...
 

Agreed, more or less. Since the problem is manifesting itself at the 1st upstream router, the problem, statistically, is local to the facility. That is, unless they do physical layer shunting across the US, which is very costly and make little or no sense.

The problem could be a myriad of things from a bad network interface on the server to a HW or SW issue on the upstream device (router or switch) or internal DNS configuration issue or similar.

There is always the possibility that the server is running IP chains or something similar and that the config is screwed up in some way, but that's way far fetched base on what I know.

Like you, I have load of network engineering experience plus telecommunications OS design, work on networking standards committees, and a prior life as a network programmer and college professor. That doesn't mean that either of us knows squat about this problem, however.

Re: Technical (internet) help PLEASE...
 

This could be the start of a networking banter thread :ss

Re: Technical (internet) help PLEASE...
 

Duuuuude! :D

Re: Technical (internet) help PLEASE...
 

Holy **** you guys lost me at hello :confused: LMAO

Re: Technical (internet) help PLEASE...
 

I would contribute a few packets!

Re: Technical (internet) help PLEASE...
 

mmblz, hope some of these thoughts end up helping out with the problems that are going on. Keep us updated and we'll keep throwing random crap your way.

Re: Technical (internet) help PLEASE...
 

Julian and I talked on the phone. We think that the problem is isolated to the configuration to iptables on the server. iptables is a security program (more or less) and it seems to be logging blocked IPs that are corresponding to at least some of those having problems. He's calling the company that owns the server now.

Re: Technical (internet) help PLEASE...
 

I know this doesn't help with the solution, but it's interesting that both admins that can't log on come from the mid atlantic area and are about two hours apart.

Re: Technical (internet) help PLEASE...
 

couldn't reach the host yet.
i think the problem might have something to with csf (a program i hadn't heard of before) ;)

Re: Technical (internet) help PLEASE...
 

Interesting

http://www.configserver.com/cp/csf.html

Re: Technical (internet) help PLEASE...
 

yeah i found a readme, etc
frank and dave got blocked for typing a bad htpasswd 5 times (lame), and i found the command to whitelist them
now have to figure out why random users would be blocked, or why traceroute would trigger blocking

Re: Technical (internet) help PLEASE...
 

ssswwwweeeeeeettttttt!

Way to go, Julian.

Re: Technical (internet) help PLEASE...
 

Must have been a senior moment for them. :ss

Re: Technical (internet) help PLEASE...
 

:confused::confused:

What language is this?

Re: Technical (internet) help PLEASE...
 

It's a language the some crazy people in an asylum made up to pass the time. ;)

Re: Technical (internet) help PLEASE...
 

see if it works next time you have a chance...

Re: Technical (internet) help PLEASE...
 

no dice


julian....when the site first went live it kept prompting me for a password. I just hit continue or entered a random password. It always rejected it....could I have the same problem as Frank and Dave?

Re: Technical (internet) help PLEASE...
 

i suppose it is vaguely possible...
though i did search for your ip, etc.. but..

i just flushed out the couple of blocked ips that i didn't think were you but maybe were.
try once more :D

Re: Technical (internet) help PLEASE...
 

Julian....think we'll need to make you an honorary Network Admin when this is done.

Re: Technical (internet) help PLEASE...
 

Wohoo...it works. Whatever switch you flipped was apparently the right one. Thanks :tu

Re: Technical (internet) help PLEASE...
 

problem solved!

Re: Technical (internet) help PLEASE...
 

thank GOD :tu

Re: Technical (internet) help PLEASE...
 

sorry to be a pain. I know you guys have been balls to the wall for a while now. This has turned into one hell of a place in just a short while. You guys should be proud. I'm def grateful :tu

Re: Technical (internet) help PLEASE...
 

no pain at all - it was the overly protective csf settings that were a pain ;)