The number you are calling FROM begins with the following: |
56K Capable |
33.6 Capable |
849, 923, 932 |
923-5432 |
923-5649 |
895 |
849-3260 |
849-2138 |
967 |
932-5508 |
932-5523 |
1/13/98: We have been moving users over to use the available capacity on 923-5649. We are still getting a peak time of usage of about ½ hour to 1 hour each night on the 923-5432 line. We have already ordered additional lines for the 923-5432 line but don't know how long it will take to have them installed. This is the same type of line we were waiting to get 4/22/97 (see below). We will continue to move users over to the 923-5649.
1/5/98: To avoid busy signals, you should know how we have the network set up. We have 2 different ways that you can dial into Cumberland Internet. Note: This information is only for people who are dialing FROM an 849-, 923 or 932 number. You can dial the 923-5432 number and it is the 56k capable line. You can also dial the 923-5649 number and it is capable of 33.6 connect speeds. To see how many modems are in use at any time on each line click on the links above. For example, the 56K capable line has a capacity of 48 modems and has just started to hit it's peak beginning Sunday, 1/4/98 between 20:00 and 22:00 hours. If you click on the 923-5649 link above, you will see that it has a capacity of 16 lines and had a maximum of 5 modems in use at the same time (1/4/98 20:00 and 22:00 hours). There were 11 lines available on the 923-5649 line. We are telling you this so you can dial which ever line suits you best. If you want to change the number you are dialing to the 923-5649 number, click Here to get instructions on how to make the change permanent.
12/7/97: E-Mail server was down from 11:00 a.m. to 8:00 p.m. - No mail was lost or deleted. You were not able to connect and send and receive mail.
12/5/97: We are adding telephone lines to the network. This should take about 1 week and we will have plenty of capacity for quite some time.
12/4/97: 2:00 a.m. - 8:30 a.m.- Consolidated's T-1 is not working properly. This is on the telephone side of the connection and is not the routing stuff like we had at the end of October. They are looking over the line to find out what happened. Update: GTE found a bad repeater in their Central Office in Greenup. Way to go Scott for finding it!!!
10/31/97 4:35 p.m.- : *sigh* Down from 4:17 pm to 4:35 pm. Hopefully just a final tweak by Consolidated to the routing. Consolidated does appear to be multi-homing (routing to multiple destinations) correctly. You can inspect their work from a DOS prompt on your own machine with a command named tracert, affectionately known as traceroute.
If you go to a DOS prompt and type the following command string
tracert www.mci.net
you'll get a nice little string of replies from every machine between us and www.mci.net. Compare the route it takes with, say,
tracert microsoft.denver.mci.net (MCI's router gateway to Microsoft)
and you should see that the route you take leaves Consolidated (which will show up as 208.132.96.33, since they don't name their routers, unlike everyone else) through a different gateway to get to the outside world, since that gateway is "closer" in router-world. That's multi-homing.
Now if we can just get Consolidated to stay up long enought to use it......
10/31/97 11:55 a.m.- : Well, we finally got a call, and Cosolidated Communications has explained exactly what happened. So that this all makes sense let's get a little background on routers and routing, and what it means to you. Trust me on this one, you do care.
Cumberland Internet is connected to Consolidated Communications via a single T-1 line (1.54 megabits per second transfer rate). Our T-1 terminates into a CSU/DSU, which is a device that converts T-1 signaling into useful bits, and then passes those bits through a V.35 serial connection to our Cisco 2501 router. Our router does one thing only and that is route network packets either into our network, or shove them out the T-1 to Consolidated. Since there is only one route (to Consolidated) our router doesn't have to think about it too much, so it is very fast and very simple. Either out to the world, or into RR1.
At Consolidated things aren't that simple. Consolidated is connected to MCI's internet backbone in three different places (Kansas City, Downers Grove, and Willow Springs). Their router is a Cisco 7000, which is a great big honking box that costs way too much money because it is one of the few routers that can accept input from a bunch of different connections downstream, and route coherently to a bunch of different connections upstream.
The Cisco 7000 uses a routing protocol called BGP (Border Gateway Protocol). BGP allows a router to receive information on routes from upstream sites, and then make decisions on what direction to send packets based on that dynamically updated information. This way a router in Central Illinois can find out about a backhoe that dug up a fiber optic line in Michigan (which did happen yesterday, but didn't affect us directly), and not even bother sending packets down that route, but will send packets around the break in the line, since it heard from other routers that that route is dead.
Now, Consolidated decided that since their router had three different routes out to the outside world that it should use them to speed up the throughput of traffic, and increase network reliability, by using BGP to make all three of those routes to MCI available to any customer coming from downstream (like RR1.net, advant.com, effingham.com,and others) instead of just static routing, like sending all our traffic through Kansas City. This was a good idea, in theory, but as you can imagine, implementation can be tricky.
On the night of the 27th, Consolidated updated the operating system software in their Cisco 7000 to a version that would allow them to do the BGP routing the way they wanted to. No problem there, except the upgrade did not go smoothly, required a couple of resets of the Cisco, and knocked us out for an hour or so. OK, we'll trade an hour of connectivity for increased speed and reliability, right? Not so fast, said the Cisco.
See the Cisco configuration that they implemented had a slight problem. When it went to communicate with the routers upstream, it apparently wasn't configured to say "please" correctly, or it barged in and demanded info it had no right to, or it spewed info out that the rest of the routers on the internet disagreed with, so the upstream routers rejected it's attempts at BGP routing. Now we still had static routing, but a lot of the processing power and time that the Cisco 7000 had available was now being used in frantic negotiations being met with brutal rejections from the routers upstream. At this point the router starts "dropping" network data packets, because it is overloaded. Very bad news for us, as this packet dropping results in what looks like everyone, including dial-up customers, getting their bandwidth cut in half, when it works at all. Severely ugly networking disorder. Working, but capital B broken.
At this point the network engineers at Consolidated made a common mistake. They thpught they just needed to tune one or two little things, and everything should work just dandy. And it should. Unfortunately, Cisco's are complicated, BGP is complicated, the manuals for BGP and Cisco's read like doctoral dissertations by Electrical Engineers, and the phone keeps ringing with some howling maniac from Cumberland something or other. You know how when there is a plane crash, and they release the transcripts of the black box, there's usually two cockpit officers screaming their heads off because there's a mountain right in front of the airplane, and one pilot is saying "Come on baby, just a little higher, you can do it.." right up to the point of impact? That pilot who still thinks they can make it was the Consolidated guys. Noble effort, trying as hard as they can, but it's still a flaming wreck on the side of the mountain.
Now, I'll give them credit. Yesterday they decided their Cisco was hosed, and they then did a very smart, and very expensive thing. They called in Cisco, and paid what is probably a a stunningly obscene amount of money (since they got you right where they want you) and got a field consultation. And last night about 10:30 p.m., a Cisco engineer, while working with the network engineers at MCI, reconfigured Consolidated's Cisco 7000 to interoperate correctly with the outside world, and at that point, our connection started to flow like butter.
So, now we have a connection to Consolidated that is running better than ever, and is able to exploit it's multiple connections to the outside world for speed and network redundancy. Now if the MCI building in Kansas City burns down the only way we'd notice would be on the nightly news.
And to their credit, the Consolidated engineers did fess up to what was the actual problem (at least they did to me, though it appears advant.com has been, well, misinformed). And for that good deed I am now paying them back by laying out their sordid tale of misadventures with BGP to you (ain't I a stinker?). So we'll watch the connections over the next few days, but I think we're out of the woods here, folks.
If any of you have any interest in the details of BGP, here's a BGP Page for you to try. And here's a link if your interested in the Cisco 7000. Pretty dry stuff, but that's the nature of routing; months of boredom, punctuated by moments of sheer terror. Just ask the engineers at Consolidated.
10/31/97 1:30 a.m.- : Things appear to have returned to an acceptable level of performance. From what can be surmised (since no one ever tells us anything), Consolidated Communications massaged their router configuration on the night of the 27th. Apparently something in their router configuration did not sit well with their upstream provider, MCI. When Consolidated got their router squared away MCI's router decided that communicating with Consolidated wasn't a very high priority (routers attached to the main backbones have policies that go something like "You're not making any sense, I'm not going to listen to you anymore").
As near as we can guess (again, guessing since no one tells us anything, since we are just customers), of the 72 hours of service varying between palsied and non-existent, it looks like we can give credit for 36 hours to Consolidated, and 36 hours to MCI, so it's a tie. (UPDATE: We have a winner. See above.)
So the next time you see an MCI ad touting the speed of their internet network, please try this to make yourself feel better; point to the TV, and in a loud clear voice, sneer "Yeah, right". (UPDATE: Still feel free to do this, since MCI can be pretty inconsistent at times, and never explains anything to anyone, period. Not even a "Whoops, sorry".)
As for Consolidated, well, today is Halloween, teenagers can get tense without their internet connections, and I understand dried egg can be cleaned off with vinegar and water.
Just kidding. 3 days of talking to people who are not in charge of apparently anything but answering the phones, saying they are working on it, and aren't authorized to get anyone in charge of anything more technical than a rotary phone, all while averaging about 4 hours sleep a night has made your system administrator a wee bit cranky. And if anyone knows their congressman personally, ask him about telecom deregulation the next time you see him.
10/29/97 11:03 a.m.- : Consolidated is currently sort of up, showing up to 30% packet loss at certain times. We still have not been contacted as to the cause of last night's outage. Industry analysts say that internet service will not become integrated in people's lives until service providers (like us) become as reliable as telephone companies (like Consolidated Communications). Pefect example of how industry analysts sometimes don't have any idea what they are talking about.
10/28/97 10:32 p.m.- : Consolidated is up again.
10/28/97 10:28 p.m.- : Consolidated is down again.
10/28/97 10:16 p.m.- : Consolidated is up again.
10/28/97 10:10 p.m.- : Consolidated is down again.
10/28/97 10:09 p.m.- : Consolidated is up again.
10/28/97 10:03 p.m.- : Consolidated is down again.
10/28/97 9:25 p.m.- : Consolidated is up again.
10/28/97 5:51 p.m.- : Consolidated is down again. Generally, they are still having problems from yesterday it appears. When will they have it figured out? We don't know. Are we looking for something else Have been for 5 months.
10/27/97 10:49 p.m.- 11:49 p.m. : Our T-1 Provider, Consolidated Communications, died temporarily when MCI had a spike outage in it's service. Also, their paging service was affected yesterday as well as part of their 800 service (like the number you call to report a problem with the pager).
10/15/97: 10:00 am - 11:10 a.m. T-1 was down. Our T-1 provider yadda yadda yadda. We'll expect no explanation but we'll post it if we get it.
10/3/97: Our T-1 provider, Consolidated Communications, was doing maintenance again last night. Once again, they failed to bring Cumberland Internet back online at the end of maintenance. We are trying to get this outage explained, but they're a phone company, so instead of solving the problem and giving us an answer they asked whether maybe we had a power outage (Of course we didn't, and we've got 200 lbs. of battery backup anyway). I'll expect a straight answer from them shortly after pigs fly.
09/19/97: Our T-1 provider, Consolidated Communications, was going to be down for a fraction of a second last night for maintenance. Evidently it did not go well as they were down from about 1:30 a.m. until about 9:30 a.m., I don't have any more information as to why or what went wrong but you can see the down time by looking at the link above labeled, "Traffic Analysis on T-1 Line to CCINet".
07/02/97: Since we have our new phone lines in place, if you are having any trouble connecting or staying connected to our network please e-mail me at Suggestions@rr1.net. The modems that are a combination Modem & Sound card are the ones that appear to have the difficulty connecting or staying connected. Please let me know if you are in this group. One thing everyone can do to connect faster is to double click on "My Computer", Double click on "Dial Up Networking", Click once on "Cumberland Internet or Internet Signup as the case may be" so it turns blue, click on "File" choose "Properties". Click on TCP/IP and under "Advanced Options" Take out the check mark in 'log on to network'. The way you take the check mark out is to click 'once' where the check mark is and it will be removed. Click on OK twice and close all windows. You are ready to connect, this will connect you with us about 5 seconds faster.
07/01/97: Beginning at about 7:30 am, MCI's Internet backbone that provides us Internet access went down in a big way. Their problem propagated into a larger problem when the switched to their backup hardware and there was at least one hardware failure and one software failure. We have lots of company with us (at least 400 other networks- maybe more). We do not have any time that this will be fixed as MCI is not giving out estimates for repair time. As you can see from Network News entry on 1/2/97 this happened before but it was for 20 minutes. As of 3:05 pm: MCI is still working on the problem. We will update this as we have information to post. 11:00 pm Update: Nothing new to report except that the MCI problem has affected their 800# service, Long Distance Service and the Internet. We are (unfortunately) low on the totem pole of problems. Our T-1 routes to Consolidated Communications and they have no information other than to wait until MCI gets their hardware lined up and settled down. Hopefully with the calm of traffic in the overnight hours that they will be able to get things working. I will post updates as I have them. 07/02/97 7:00 am Update: We are up an running and appear to be normal. I would expect that our connection to the Internet today may be unstable today but at this time it appears to be o.k. - Our T-1 that routes through Consolidated has 2 direct connections to the Internet (Independence, KS & Willow Springs, IL) and both connections were affected by yesterdays problems at MCI. I will report back as I have more. MCI came back up at about 1:00 am on 7/2/97. Here's a link that mentions (briefly - at the bottom of the page) the problem MCI had on 7/1/97.
06/27/97: Server was not authenticating users from 10:45 am to 3:00 pm- We have changed how the servers authenticate and both servers back up each other now. This should solve this problem for the time being. Both servers would have to go down now for this problem to occur again.
05/25/97: Server was not authenticating dial in users from 1:10 pm to 1:58 pm We restarted the server and are back to normal.
05/21/97: We were adjusting the software that runs the e-mail and it was giving error messages between 5:45 p.m. and 7:45 p.m.- We fixed the error.
05/09/97: The computer that runs the news server and the back up for the webserver is down. We are working to fix it but don't expect it to be back up before Tuesday 5/13/97. In the mean time, if you got to this page, you have figured out that you can go all over the web but the DNS look up for www.rr1.net for internal users is timing out and you are getting the message web site not found. To get to the home page type http://208.132.114.5 in the address line and hit enter. Book mark that address or change the start up page in your browser to this page to avoid the problem when you log on. All the links off the home page have been changed to reflect the DNS rather than the www.rr1.net address and should work. E-mail is working locally but it is slow from outside. We will post updates as we have information. For those people outside RR1.net trying to look at the rr1.net homepage it is coming up just fine on their network. Please let me know if you have any questions and thanks for your patience. By the way, if you want to download a program that will let you 'chat'/e-mail with the outside world to selected users, download the program ICQ, click here to got to the page that explains the program and to download it. Back to normal 5/12/97 7:00 p.m.- We are making provisions to make sure the same problem does not occur again and sorry for the disruption.
04/22/97: We have not been signing up new subscribers for the past 30+ days. We have a waiting list and will sign up all users on the waiting list as soon as we have our digital phone lines installed and properly tested. We estimate that this sign up date will be May 3, 1997 (based on the latest installation date promised to us by GTE). We did not want to sign on new users without having the line capacity on our network and give poor service (i.e. busy signals) to our valued subscribers who have become very used to service with no busy signals. Our new phone lines will allow us to handle 56kbps modems and we will post more about this as we get closer. I have had questions from people about our waiting list and I wanted to explain the situation. Please let me know if you have any questions at Suggestions@rr1.net
4/3/97: The T-1 which connects us to the Internet was down beginning at 1:39:27.3 p.m. today. We have our provider working on it and will have it up as fast as we can. We will post updates to this page as we have them available. Update: the T-1 was down until 3:58:43.1 p.m. Thursday 4/3/97. The downtime was due to a switch that reset (MUX Circuit) at Illinois Consolidated Telephone Company. When this circuit reset, our routing was not reset in automatically (it is NOW!). In the end, we did have one benefit of the downtime. Consolidated took advantage of our downtime to switch us from copper lines to fiber lines from Greenup to Neoga. Our line to St. Louis is now all fiber with the exception of the line that comes across the alley from the Central Office in Greenup to the back of the Radio Shack Building.
3/28/97: The power was out in Greenup at the Radio Shack from 5:55 P.M. to 7:25 P.M. due to strong storms in the area. Our back up power system kept the network in operation until 7:22 P.M. so we were 3 minutes short of covering the entire power outage. The Village of Greenup reports that the power may be intermittent until they get all the power back on in the village. Our back up power is recharging and hope you will not be affected by this as a network user. That's all for now, check back for updates.3/29/97 update: The power in the Village of Greenup is stable now.
3/15/97: We have done some work on our Mail Server that caused the mail to not be delivered between approximately 5 p.m. Friday to 2 p.m. Saturday. We were attempting to add a filter on the mail server and it held up delivery of all mail. The mail server should be back to normal once it processes all the backlogged mail.
3/2/97: The company that provides our Internet Connection will be running a test on our T-1 line that will take between 10-30 minutes. They will begin the test sometime between 12:30 p.m. and 1:30 p.m.. During this time you will be able to connect to us, check e-mail, look at local web pages (on our server), and visit newsgroups. You will not get a response if you try to use the world wide web for sites off our server.
1/31/97: We will have the network down for about 1 hour beginning at approximately 2:00 p.m. to update software. (We have determined that this time of day has the minimum number of users, we don't want to cause an interruption during high demand time). Update: We brought network down at 2:48 p.m. and came back up at 3:40 p.m.- I appears our updates are working correctly. Shortly after we came back online, we noticed our Windows 3.1 users could not authenticate and logon. We checked out the system and made the necessary changes and Windows 3.1 users could logon around 6 p.m.- sorry for the inconvenience. We will have to make routine upgrades to the hardware & software. We'll post when we have to do this again. Thanks.
1/14/97: Additional phone lines will be up and running by Friday, January 17th. We thought we would have these lines up by the end of last week but they should be in place by the end of this week. Sorry for any logon problems caused by busy signals. We strive to have the necessary lines in place prior to demand exceeding supply. Update: Additional lines are in place and operating correctly.
1/2/97: MCI backbone network was down for about 20 minutes. They are a major backbone for Internet and caused a loss of service on rr1.net for the same period. You could connect to our machine but could not navigate. It affected about 30 million others at the same time so you had company.
12/28/96: The system was down from 10:55 p.m. to just before 1:00 a.m.- We will have a scheduled maintenance in the next 72 hours to update some server software to correct this unscheduled downtime.
12/27/96: One of our phone lines in our hunt group is not routed properly by GTE. This will result in a ringing and no answer by the network when this line is hit in the hunt group. We should have this corrected on Monday 12/30/96. UPDATE: The line is updated. It was not GTE it was a bad wire in the patch between the modem and the serial card.
If you have any questions, comments, etc. e-mail suggestions@rr1.net.