|Tools: Mailman - Time Online - Chat Room - News Ticker - eTools - Network News - Order CD|
|Services: Online Classes - Free Classes - Message Board - Classifieds - Online Help - Bridge Camera|
|About Us: About Us - New User Info - FAQs - Account Change Form - Tech Info - Fast Wireless Internet|
|Local: Email Directory - Webpages - Shops - Movies - Golf - Pics - Gov't - Gov't Minutes - Discuss|
|Starting Places: Excite - Yahoo - Hotbot - Go - Lycos - About.com - Looksmart - Altavista - Search - Google|
|Cool Stuff: Auctions - Agriculture - Genealogy - Health - Sports - News/Money - Software - eTools - Ask.com|
10/25/99 4:22 p.m.- We have an open ticket with GTE on the problems with our dial in T-1 lines. All calls going to the first 2 T-1's are not being connected to our modems. We have disconnected those two dial in T-1's from our equipment, and all calls are now being answered by our other T-1's(currently 96 total modems available). GTE is working on it, and we'll keep you informed as soon as we hear anything from them. 5:35 p.m. - All 144 phone lines are up now and appear to be working correctly. We'll keep a close eye on it to see what happens the next 24 hours.Thank you for your patience while we worked on this problem. Since it was intermittent, it was difficult to troubleshoot for GTE.
10/22/99 5:00 p.m.- We are seeing some problems connecting when dialing in or staying connected. Not all users are seeing it but some modems are more affected than others. We are working on this problem and don't know it is the same problem as was listed 10/9/99 or not. We are checking everything out and will update this page to let you know what we find.
10/21/99 4:25 p.m.: New users dialing in would have been prompted for their password and not connect. Existing users dialed in would not have noticed any problem. 4:35 p.m. We reset the authentication software and it is working correctly now.
10/9/99 : Our first incoming T-1 has decided that it is going to recycle itself every ten minutes or so. GTE is going to be working on it(anybody else recognize a pattern here?), probably taking that T-1 out of commission, and routing all incoming calls to start with our second T-1. That should fix things while they inspect the defective T-1. We are sorry for the inconvenience, and will keep you updated as to the status of this situation.
9/27/99 (Evening): Our incoming phone lines are causing problems. The phone line is stopping operation, resetting and then repeating the process. To you as a dialer, you would not be able to keep your connection and you would be kicked offline. When you reconnect, the process repeats. We had GTE working on it until Midnight and they saw the problem occurring. They have put some tracing equipment on the lines to figure out what is going on but we don't have the results yet. We are still working on this and hope to repair it today. Update 9/28/99 6:00 p.m.: GTE has switched our incoming T-1's (the phone lines you dial in on) over to a different Fiber Card in the Central Office and this appears to have fixed the problem of the T-1's resetting and causing disconnects.
9/19/99 (4 am - 1:30 p.m.): Our T-1 circuit is down. We have been working on it since 7 am. You can go to anything that is stored locally but not outside. We will post with an update as soon as we have one available. Update: There is a phone circuit between Indianapolis and St. Louis that is down and it is one we use to connect to the Internet. We have called everyone possible to get this phone line repaired but we don't have an estimated time yet on when it will be up. 12:21 p.m. McLeod is going to turn on an alternate line from their Indianapolis Switch to the local router. They have quit trying to fix the old line and are replacing the line. Normally it takes about 2 months to get a T-1 installed but they are doing it now, we hope it will be up around 3 pm. 1:30 p.m.- we're back up. We're sorry for the inconvenience.
9/6/99 (2 am - 10:30 am): The 33.6 modems were not answering the phone if you dialed into them. This is the 2nd time this has happened at 2 am so we are looking at what is causing the modems to all get reset at the same time. This is why every user should know their alternate phone number to dial in if their primary number is unavailable. Click on the link above labeled 'Alternate Phone Line Page' to make a note of your back up phone number. Again, we are working to determine what caused this problem to keep it from occurring again in the future. The 56k modems were working fine during this entire period. Update: The problem was because of spikes in the incoming power. We have repaired the problem.
7/11/99: (6:00 a.m- currently): Looks like the same problem from 7/9/99- Unable to dial in or when you can dial in you get to our homepage but not outside. We're working on it, watch for updates.
7/9/99: (4:30 p.m. - 5:40 p.m.): Correction: Telephone Circuit was down. You can get to the Cumberland Internet Homepage but not outside. Update: Telephone circuit repaired at 5:40 pm.
7/1/99: (3:24 p.m. - 4:20 p.m.): Power problems in Greenup. Power was up and down while 1/2 of the 220 power was not working in Greenup. All power turned off by Greenup to correct problem. You could connect off and on in this period. At the moment, you can connect and get local pages but not the outside pages. We'll fix it a.s.a.p., watch here for updates. (Update 4:45 p.m. - It's all working now)
6/27/99: (2:15 p.m.- 3:45 p.m.): You can dial in and get local pages but the route out to other pages on the Internet is not working. We are working on it right now and will post an update as soon as we have it available. (Update 3:35 p.m.: We have identified the router that is the problem and are working on a solution. It should not be too much longer. Watch for further updates here.) 3:45 p.m.- We're back up and working again. It looks like it was an ethernet port on a router that died. Sorry for any delay or problems.
6/10/99 (9:59 a.m.): If you are having any problem with an auction on eBay it is because eBay is having some serious problems on their end. They've had a hardware failure and have suspended auctions. You can read about it here. You can also go directly to the eBay network news page that they update often.
6/7/99: Cumberland Internet has another connection to the Internet (see above chart T-1 line to UCIX). What this means to you is that your webpages will load faster. This is because your requests are routed more efficiently through lines that have less traffic than the McLeod T-1. Cumberland Internet is your FAST way to the Internet. Enjoy!
6/6/99: If you have a counter on your page, e-mail email@example.com to give me the correct page count. These were reset when we updated our harddrive from the 6/2/99 incident below.
6/2/99 (~2 am - 5:05 am): The Server (Computer) that handles all e-mail for Cumberland Internet had a hard drive crash around 2 a.m.. By 5 a.m. 6/2/99 we had restored mail services and were able to send and receive e-mail to rr1.net users. However, the mail that was on the server at the time of the crash is not likely to be recovered. We will make every effort to recover the data from the damaged hard drive but the bottom line is that if you have sent e-mail to a rr1.net user in the last 24 hours, you might want to resend it to that user because they might not have gotten it. If you regularily e-mail to other people, you might let them know to resend you any mail that they might have sent in 6/1/99. We apologize for any inconvenience and will strive to keep you fully informed of anything that affects your service
5/13/99 ( 11 pm) to 5/14/99 (1:00 pm): All WebTV's servers were down. All users who were using a webtv box were unable to connect to the Internet. This affected every WebTV user in the United States and not just WebTV users who were dialing Cumberland Internet. If you have questions or want additional information, feel free to call 1-800-GOWEBTV.
5/12/99 (~4:00 p.m.) to 5/13/99 (10 a.m.): You would not have been able to connect to our FTP server. It is now up and working correctly.
3/28/99 (9:15 am - 10:00 am): We had to reset the mail server. You were unable to receive your mail during the time listed. You were, however, able to send mail during this time.
2/2/99 (Midnight - 5 am): System maintenance on the mail server. You were able to log on and go anywhere on the web, but you were not able to get your mail during this period.
1/29/99: This information is about the status of getting Newsgroups. Our Newsgroup feed is from Consolidated/McLeod Communications. Their main server for handling news died and all the traffic is being handled by a small, backup server. This small, backup server cannot handle all the traffic for the news feed. They are sending out a smaller, newsfeed in the interim. They have a new server that they can put online but before it is allowed to go online, they must cerfify it for Y2K. That should take until the end of next week. Our estimated date for a full newsfeed is February 5, 1999. You can still read newsgroups by going to www.dejanews.com until the full feed is returned. Please let me know if you have anyquestions.
1/19/99: Westfield Users Only- The lines that you call on are water logged and have a scratchy sound that will interfere with your connection. We have called GTE to fix this and we hope it will be fixed tomorrow 1/20/99.
1/13/99 (10:30 am -Noon): CIPS Power grid in Greenup was down starting shortly after 9 am until noon. Toledo, Greenup and Casey had no power while Neoga still had power. Neoga users were able to continue online until about 10:30 am when the back up batteries were used up.
12/06/98: Greenup Evening Power Flash until 10:24 pm- Homepage would not load. We moved the server that loads webpages to the Uninterruptable Power Supply (UPS). We thought we had it plugged into this already but....Our mistake!
11/30/98: 9 pm -10 :30 pm (off and on): The homepage may have not been available during the time mentioned. The page was available off and on during that time. We were moving WWW and FTP services to a different (faster!) server. We apologize for any inconvenience. Also, any of you who are using FTP to upload the webpages, you will need to upload to ftp.rr1.net rather than to the old domain number (18.104.22.168). If you have questions, check out our FTP Upload page. Also, the scripts for updating your webpage counter should be as follows (The only difference is a lowercase 'S' in scripts: <CENTER><FONT SIZE=+1>You are visitor number <IMG src=/scripts/counter.exe?-w5+yourusername.htm> since xx/xx/98<BR> </FONT></CENTER>
11/19/98: 4:30 pm- 5:10 pm- Consolidated Communications is transferring all Internet handling over to McLeod USA in Iowa from their site in St. Louis. Their routers are currently behaving in a manner that could be described as broken. Evidently even Consolidated Communications doesn't even know how to reach the Internet Network Operations Center in Iowa. Really, I couldn't make up this good of a story. As it stands we have a trouble ticket open with the phone company side of the business and we are hoping that they can find the Internet Network Operations Center phone number in Iowa soon. In the mean time you can get to our pages but anything outside is slllllooooowwww. Update: 7:30 p.m.- we got a hold of the Network Operations Center in Iowa and they are looking at rerouting our T-1. The volume of traffic on the route we are taking through Consolidated (in St. Louis) is starting to cause packet loss problems.
10/20/98: 9:00 p.m. - 11:00 p.m. - 2 of our 4 incoming T-1 lines that are the 'phone lines' that you dial in on when you are calling the 56k modems were having frame errors. We contacted GTE and they worked on the problem. What happens with this error is that you will call our 56k line and get nothing. If you pick up the telephone and call it you will dial the phone and then get nothing. If you retried, there were enough non-frame errors that you would get through after repeated attempts. Basically, the incoming phone lines were not connecting through the central office successfully.
8/6/98 9:45 a.m.-11:00 a.m. - Our T-1 circuit to Consolidated was down. I still don't have a satisfactory explanation from them yet. Consolidated ran a test sequence on the circuit and it came back up after the test. Of course they only did that after having us reset every piece of equipment we have, to make sure it's not us causing the problem. THEN they test their stuff. And that fixes it. Consolidated says it's not clear what the problem is, and can't say for sure where the error occurred. We disagree. We are of the opinion that a specific piece of equipment (just guessing, but maybe a frame switch) has a problem with resets. We've seen this before, but they didn't fix it on just our word last time, so it looks like your friendly neighborhood system administrator gets to sleep next to his computer for a week or two.
7/29/98 12:00 a.m.-4:00 a.m. - GTE will be updating the wire circuitry that supports our T-1 internet connection to Consolidated. There might be interruptions, intermittently, to our upstream connection. Most of the time these upgrades are done with only a minute or two of downtime at the most, so you might not even notice this outage if things go well. Cross your fingers.
7/27/98 7:15 a.m.-10:15 a.m. - MCI updated their routers this morning, and mis-configured the routing for the ports that lead to Consolidated Communications. As a result all customers of CCI high speed internet services (us, advant.com, effingham.com, and at least 18 others) were unable to access the internet. MCI corrected their routing tables at 10:15 and things are hopefully back to normal.
7/24/98 ~1:45 p.m.-5:15 p.m. - Invasion of the backhoes! A slightly important piece of fiber optic cable was cut south of Mattoon on Rt. 45. Apparently the fiber splice crew jumped right on it since we had a repair update that it would be done at 7:00, and we were back up at 5:15. Must have had telephone traffic on it. ;)
Now you know why all system administrators with wide area links breath a sigh of relief when construction season ends.
7/18/98 10:30 p.m.-12:00 p.m. - RR1.net was the victim of a Denial Of Service attack. Some creative little computer user out on the internet apparently had a little spat with another user. The creative user decided to attack his enemy using a specific component of the IP protocol, causing our network and 5 or 6 other networks to politely inquire as to the network status of the victim's network several thousand times a second. This clogged our network pipes, as weel as the other networks, and of course the intended victim got deluged. Once we were able to identify the source of the problem we modified our router configuration with a filter on these specific types of attacks. Our upstream provider also modified their router configuration to also provide another layer of protection. These modifications should prevent a reccurence of the same type of attack.
Just another example of a formerly useful network service becoming a point of abuse by immature and destructive network users (similar to what has happened with spam and e-mail). Of course, this type of attack is specifically illegal, and if this network abuser is tracked down, suffice it to say 5 or 6 ISP's would be very interested in pressing charges. Then the fun starts, because the penalties for Denial-of-Service attacks include fines and (drum-roll, please) jail time.
Just a tidbit of information for anyone else who thinks they are a clever network user. As was stated by an RR1 user earlier this week, "there is no internet access in Marion".
6/16/98 10:15 a.m.-10:30 a.m. - GTE did preventative maintenance on our T-1 that connects us to the Internet. You could still dial in and get local webpages but not others (like infoseek or others).
6/10/98: 3:30 p.m. - The T-1 line that you dial in on with the 923-5432 line is down. We are working with GTE to get it fixed. If you call the alternate number (follow link above), you'll get the alternate number for your calling area. Update: 3:54 p.m. The primary T-1 is still being fixed but you can connect calling 923-5432 because you are calling the 2nd and 3rd T-1 line. Thanks Scott & Phil for getting us up so quickly!
5/30/98: 9 am-Noon- You again were not able to browse the rr1.net homepage. You were still able to browse the Internet if you went to other pages. Repeat of the 5/4/98 issue. Update: 5/31/98 9:30 am- Web page counters are not showing up on local web pages. Don't change anything on your webpage, we will fix it so the counter image is working shortly. I will still keep count of your hits the graphic is just not loading.
5/4/98: 8:45 p.m. - 9:15 p.m. -You were not able to browse the rr1.net homepage because we were updating our server software. You were still able to browse the Internet if you went to other addresses, just the homepage was unavailable.
4/13/98: One modem on the 33.6 modems was not picking up correctly on Easter Sunday. You would have gotten a busy signal or a ringing with no answer. It is repaired now.
4/11/98: The modem status and server status graphs are hosted on a remote site that is not up right now. Cumberland Internet is up and running fine, you just can't get the graphs right now. 8:30 p.m. Update: The graphs are available again.
2/13/98: We are not adding new users until we get the T-1 line we ordered over 1 month ago delivered from GTE. When this is installed the 56k capable line will have 72 lines available. We held up adding new users 4/22/97 waiting for the same type of line to be installed. Update: We have an installation date of March 7 from GTE for the additional T-1. We expect to test the line after it is installed and then post a notice here that we will be taking new users again.
1/31/98: All users can now connect to either modem usage graph listed above. The 56K capable line is the one that is reaching capacity (and we have ordered additional capacity). The 33.6 capable modems can be reached by ALL Cumberland Internet users and has plenty of capacity. To change the number on a one time basis, just click and type in the number directly on the screen that has your username and password prior to connecting. To make your change permanent, click Here for instructions. Please e-mail Suggestions@rr1.net if you encounter busy signals on both line choices.
1/13/98: We have been moving users over to use the available capacity on 923-5649. We are still getting a peak time of usage of about ½ hour to 1 hour each night on the 923-5432 line. We have already ordered additional lines for the 923-5432 line but don't know how long it will take to have them installed. This is the same type of line we were waiting to get 4/22/97 (see below). We will continue to move users over to the 923-5649.
1/5/98: To avoid busy signals, you should know how we have the network set up. We have 2 different ways that you can dial into Cumberland Internet. Note: This information is only for people who are dialing FROM an 849-, 923 or 932 number . You can dial the 923-5432 number and it is the 56k capable line. You can also dial the 923-5649 number and it is capable of 33.6 connect speeds. To see how many modems are in use at any time on each line click on the links above. For example, the 56K capable line has a capacity of 48 modems and has just started to hit it's peak beginning Sunday, 1/4/98 between 20:00 and 22:00 hours. If you click on the 923-5649 link above, you will see that it has a capacity of 16 lines and had a maximum of 5 modems in use at the same time (1/4/98 20:00 and 22:00 hours). There were 11 lines available on the 923-5649 line. We are telling you this so you can dial which ever line suits you best. If you want to change the number you are dialing to the 923 -5649 number, click Here to get instructions on how to make the change permanent.
12/7/97: E-Mail server was down from 11:00 a.m. to 8:00 p.m. - No mail was lost or deleted. You were not able to connect and send and receive mail.
12/5/97: We are adding telephone lines to the network. This should take about 1 week and we will have plenty of capacity for quite some time.
12/4/97: 2:00 a.m. - 8:30 a.m.- Consolidated's T-1 is not working properly. This is on the telephone side of the connection and is not the routing stuff like we had at the end of October. They are looking over the line to find out what happened. Update: GTE found a bad repeater in their Central Office in Greenup. Way to go Scott for finding it!!!
10/31/97 4:35 p.m.- : *sigh* Down from 4:17 pm to 4:35 pm. Hopefully just a final tweak by Consolidated to the routing. Consolidated does appear to be multi-homing (routing to multiple destinations) correctly. You can inspect their work from a DOS prompt on your own machine with a command named tracert, affectionately known as traceroute.
If you go to a DOS prompt and type the following command string
you'll get a nice little string of replies from every machine between us and www.mci.net. Compare the route it takes with, say,
tracert microsoft.denver.mci.net (MCI's router gateway to Microsoft)
and you should see that the route you take leaves Consolidated (which will show up as 22.214.171.124, since they don't name their routers, unlike everyone else) through a different gateway to get to the outside world, since that gateway is "closer" in router-world. That's multi-homing.
Now if we can just get Consolidated to stay up long enought to use it......
10/31/97 11:55 a.m.- : Well, we finally got a call, and Cosolidated Communications has explained exactly what happened. So that this all makes sense let's get a little background on routers and routing, and what it means to you. Trust me on this one, you do care.
Cumberland Internet is connected to Consolidated Communications via a single T-1 line (1.54 megabits per second transfer rate). Our T-1 terminates into a CSU/DSU, which is a device that converts T-1 signaling into useful bits, and then passes those bits through a V.35 serial connection to our Cisco 2501 router. Our router does one thing only and that is route network packets either into our network, or shove them out the T-1 to Consolidated. Since there is only one route (to Consolidated) our router doesn't have to think about it too much, so it is very fast and very simple. Either out to the world, or into RR1.
At Consolidated things aren't that simple. Consolidated is connected to MCI's internet backbone in three different places (Kansas City, Downers Grove, and Willow Springs). Their router is a Cisco 7000, which is a great big honking box that costs way too much money because it is one of the few routers that can accept input from a bunch of different connections downstream, and route coherently to a bunch of different connections upstream.
The Cisco 7000 uses a routing protocol called BGP (Border Gateway Protocol). BGP allows a router to receive information on routes from upstream sites, and then make decisions on what direction to send packets based on that dynamically updated information. This way a router in Central Illinois can find out about a backhoe that dug up a fiber optic line in Michigan (which did happen yesterday, but didn't affect us directly), and not even bother sending packets down that route, but will send packets around the break in the line, since it heard from other routers that that route is dead.
Now, Consolidated decided that since their router had three different routes out to the outside world that it should use them to speed up the throughput of traffic, and increase network reliability, by using BGP to make all three of those routes to MCI available to any customer coming from downstream (like RR1.net, advant.com, effingham .com,and others) instead of just static routing, like sending all our traffic through Kansas City. This was a good idea, in theory, but as you can imagine, implementation can be tricky.
On the night of the 27th, Consolidated updated the operating system software in their Cisco 7000 to a version that would allow them to do the BGP routing the way they wanted to. No problem there, except the upgrade did not go smoothly, required a couple of resets of the Cisco, and knocked us out for an hour or so. OK, we'll trade an hour of connectivity for increased speed and reliability, right? Not so fast, said the Cisco.
See the Cisco configuration that they implemented had a slight problem. When it went to communicate with the routers upstream, it apparently wasn't configured to say "please" correctly, or it barged in and demanded info it had no right to, or it spewed info out that the rest of the routers on the internet disagreed with, so the upstream routers rejected it's attempts at BGP routing. Now we still had static routing, but a lot of the processing power and time that the Cisco 7000 had available was now being used in frantic negotiations being met with brutal rejections from the routers upstream. At this point the router starts "dropping" network data packets, because it is overloaded. Very bad news for us, as this packet dropping results in what looks like everyone, including dial -up customers, getting their bandwidth cut in half, when it works at all. Severely ugly networking disorder. Working, but capital B broken.
At this point the network engineers at Consolidated made a common mistake. They thpught they just needed to tune one or two little things, and everything should work just dandy. And it should. Unfortunately, Cisco's are complicated, BGP is complicated, the manuals for BGP and Cisco's read like doctoral dissertations by Electrical Engineers, and the phone keeps ringing with some howling maniac from Cumberland something or other. You know how when there is a plane crash, and they release the transcripts of the black box, there's usually two cockpit officers screaming their heads off because there's a mountain right in front of the airplane, and one pilot is saying "Come on baby, just a little higher, you can do it.." right up to the point of impact? That pilot who still thinks they can make it was the Consolidated guys. Noble effort, trying as hard as they can, but it's still a flaming wreck on the side of the mountain.
Now, I'll give them credit. Yesterday they decided their Cisco was hosed, and they then did a very smart, and very expensive thing. They called in Cisco, and paid what is probably a a stunningly obscene amount of money (since they got you right where they want you) and got a field consultation. And last night about 10:30 p.m., a Cisco engineer, while working with the network engineers at MCI, reconfigured Consolidated's Cisco 7000 to interoperate correctly with the outside world, and at that point, our connection started to flow like butter.
So, now we have a connection to Consolidated that is running better than ever, and is able to exploit it's multiple connections to the outside world for speed and network redundancy. Now if the MCI building in Kansas City burns down the only way we'd notice would be on the nightly news.
And to their credit, the Consolidated engineers did fess up to what was the actual problem (at least they did to me, though it appears advant.com has been, well, misinformed). And for that good deed I am now paying them back by laying out their sordid tale of misadventures with BGP to you (ain't I a stinker?). So we'll watch the connections over the next few days, but I think we're out of the woods here, folks.
If any of you have any interest in the details of BGP, here's a BGP Page for you to try. And here's a link if your interested in the Cisco 7000. Pretty dry stuff, but that's the nature of routing; months of boredom, punctuated by moments of sheer terror. Just ask the engineers at Consolidated.
10/31/97 1:30 a.m.- : Things appear to have returned to an acceptable level of performance. From what can be surmised (since no one ever tells us anything), Consolidated Communications massaged their router configuration on the night of the 27th. Apparently something in their router configuration did not sit well with their upstream provider, MCI. When Consolidated got their router squared away MCI's router decided that communicating with Consolidated wasn't a very high priority (routers attached to the main backbones have policies that go something like "You're not making any sense, I'm not going to listen to you anymore").
As near as we can guess (again, guessing since no one tells us anything, since we are just customers), of the 72 hours of service varying between palsied and non-existent, it looks like we can give credit for 36 hours to Consolidated, and 36 hours to MCI, so it's a tie. (UPDATE: We have a winner. See above.)
So the next time you see an MCI ad touting the speed of their internet network, please try this to make yourself feel better; point to the TV, and in a loud clear voice, sneer "Yeah, right". (UPDATE: Still feel free to do this, since MCI can be pretty inconsistent at times, and never explains anything to anyone, period. Not even a "Whoops, sorry".)
As for Consolidated, well, today is Halloween, teenagers can get tense without their internet connections, and I understand dried egg can be cleaned off with vinegar and water.
Just kidding. 3 days of talking to people who are not in charge of apparently anything but answering the phones, saying they are working on it, and aren't authorized to get anyone in charge of anything more technical than a rotary phone, all while averaging about 4 hours sleep a night has made your system administrator a wee bit cranky. And if anyone knows their congressman personally, ask him about telecom deregulation the next time you see him.
10/29/97 11:03 a.m.- : Consolidated is currently sort of up, showing up to 30% packet loss at certain times. We still have not been contacted as to the cause of last night's outage. Industry analysts say that internet service will not become integrated in people's lives until service providers (like us) become as reliable as telephone companies (like Consolidated Communications). Pefect example of how industry analysts sometimes don't have any idea what they are talking about.
10/28/97 10:32 p.m.- : Consolidated is up again.
10/28/97 10:28 p.m.- : Consolidated is down again.
10/28/97 10:16 p.m.- : Consolidated is up again.
10/28/97 10:10 p.m.- : Consolidated is down again.
10/28/97 10:09 p.m.- : Consolidated is up again.
10/28/97 10:03 p.m.- : Consolidated is down again.
10/28/97 9:25 p.m.- : Consolidated is up again.
10/28/97 5:51 p.m.- : Consolidated is down again. Generally, they are still having problems from yesterday it appears. When will they have it figured out? We don't know. Are we looking for something else Have been for 5 months.
10/27/97 10:49 p.m.- 11:49 p.m. : Our T-1 Provider, Consolidated Communications, died temporarily when MCI had a spike outage in it's service. Also, their paging service was affected yesterday as well as part of their 800 service (like the number you call to report a problem with the pager).
10/15/97: 10:00 am - 11:10 a.m. T-1 was down. Our T-1 provider yadda yadda yadda. We'll expect no explanation but we'll post it if we get it.
10/3/97: Our T-1 provider, Consolidated Communications, was doing maintenance again last night. Once again, they failed to bring Cumberland Internet back online at the end of maintenance. We are trying to get this outage explained, but they're a phone company, so instead of solving the problem and giving us an answer they asked whether maybe we had a power outage (Of course we didn't, and we've got 200 lbs. of battery backup anyway). I'll expect a straight answer from them shortly after pigs fly.
09/19/97: Our T-1 provider, Consolidated Communications, was going to be down for a fraction of a second last night for maintenance. Evidently it did not go well as they were down from about 1:30 a.m. until about 9:30 a.m., I don't have any more information as to why or what went wrong but you can see the down time by looking at the link above labeled, "Traffic Analysis on T-1 Line to CCINet".
07/02/97: Since we have our new phone lines in place, if you are having any trouble connecting or staying connected to our network please e-mail me at Suggestions@rr1.net. The modems that are a combination Modem & Sound card are the ones that appear to have the difficulty connecting or staying connected. Please let me know if you are in this group. One thing everyone can do to connect faster is to double click on "My Computer", Double click on "Dial Up Networking", Click once on "Cumberland Internet or Internet Signup as the case may be" so it turns blue, click on "File" choose "Properties". Click on TCP/IP and under "Advanced Options" Take out the check mark in 'log on to network'. The way you take the check mark out is to click 'once' where the check mark is and it will be removed. Click on OK twice and close all windows. You are ready to connect, this will connect you with us about 5 seconds faster.
07/01/97: Beginning at about 7:30 am, MCI's Internet backbone that provides us Internet access went down in a big way. Their problem propagated into a larger problem when the switched to their backup hardware and there was at least one hardware failure and one software failure. We have lots of company with us (at least 400 other networks- maybe more). We do not have any time that this will be fixed as MCI is not giving out estimates for repair time. As you can see from Network News entry on 1/2/97 this happened before but it was for 20 minutes. As of 3:05 pm: MCI is still working on the problem. We will update this as we have information to post. 11:00 pm Update: Nothing new to report except that the MCI problem has affected their 800# service, Long Distance Service and the Internet. We are (unfortunately) low on the totem pole of problems. Our T-1 routes to Consolidated Communications and they have no information other than to wait until MCI gets their hardware lined up and settled down. Hopefully with the calm of traffic in the overnight hours that they will be able to get things working. I will post updates as I have them. 07/02/97 7:00 am Update: We are up an running and appear to be normal. I would expect that our connection to the Internet today may be unstable today but at this time it appears to be o.k. - Our T-1 that routes through Consolidated has 2 direct connections to the Internet (Independence, KS & Willow Springs, IL) and both connections were affected by yesterdays problems at MCI. I will report back as I have more. MCI came back up at about 1:00 am on 7/2/97. Here's a link that mentions (briefly - at the bottom of the page) the problem MCI had on 7/1/97.
06/27/97: Server was not authenticating users from 10:45 am to 3:00 pm- We have changed how the servers authenticate and both servers back up each other now. This should solve this problem for the time being. Both servers would have to go down now for this problem to occur again.
05/25/97: Server was not authenticating dial in users from 1:10 pm to 1:58 pm We restarted the server and are back to normal.
05/21/97: We were adjusting the software that runs the e-mail and it was giving error messages between 5:45 p .m. and 7:45 p.m.- We fixed the error.
05/09/97: The computer that runs the news server and the back up for the webserver is down. We are working to fix it but don't expect it to be back up before Tuesday 5/13/97. In the mean time, if you got to this page, you have figured out that you can go all over the web but the DNS look up for www.rr1.net for internal users is timing out and you are getting the message web site not found. To get to the home page type http://126.96.36.199 in the address line and hit enter. Book mark that address or change the start up page in your browser to this page to avoid the problem when you log on. All the links off the home page have been changed to reflect the DNS rather than the www.rr1.net address and should work. E-mail is working locally but it is slow from outside. We will post updates as we have information. For those people outside RR1.net trying to look at the rr1.net homepage it is coming up just fine on their network. Please let me know if you have any questions and thanks for your patience. By the way, if you want to download a program that will let you 'chat'/e-mail with the outside world to selected users, download the program ICQ, click here to got to the page that explains the program and to download it. Back to normal 5/12/97 7:00 p.m.- We are making provisions to make sure the same problem does not occur again and sorry for the disruption.
04/22/97: We have not been signing up new subscribers for the past 30+ days. We have a waiting list and will sign up all users on the waiting list as soon as we have our digital phone lines installed and properly tested. We estimate that this sign up date will be May 3, 1997 (based on the latest installation date promised to us by GTE). We did not want to sign on new users without having the line capacity on our network and give poor service (i.e. busy signals) to our valued subscribers who have become very used to service with no busy signals. Our new phone lines will allow us to handle 56kbps modems and we will post more about this as we get closer. I have had questions from people about our waiting list and I wanted to explain the situation. Please let me know if you have any questions at Suggestions@rr1.net
4/3/97: The T-1 which connects us to the Internet was down beginning at 1:39:27.3 p.m. today. We have our provider working on it and will have it up as fast as we can. We will post updates to this page as we have them available. Update: the T-1 was down until 3:58:43.1 p.m. Thursday 4/3/97. The downtime was due to a switch that reset (MUX Circuit) at Illinois Consolidated Telephone Company. When this circuit reset, our routing was not reset in automatically (it is NOW!). In the end, we did have one benefit of the downtime. Consolidated took advantage of our downtime to switch us from copper lines to fiber lines from Greenup to Neoga. Our line to St. Louis is now all fiber with the exception of the line that comes across the alley from the Central Office in Greenup to the back of the Radio Shack Building.
3/28/97: The power was out in Greenup at the Radio Shack from 5:55 P.M. to 7:25 P.M. due to strong storms in the area. Our back up power system kept the network in operation until 7:22 P.M. so we were 3 minutes short of covering the entire power outage. The Village of Greenup reports that the power may be intermittent until they get all the power back on in the village. Our back up power is recharging and hope you will not be affected by this as a network user. That's all for now, check back for updates.3/29/97 update: The power in the Village of Greenup is stable now.
3/15/97: We have done some work on our Mail Server that caused the mail to not be delivered between
approximately 5 p.m. Friday to 2 p.m. Saturday. We were attempting to add a filter on the mail server and it held up delivery of all mail. The mail server should be back to normal once it processes all the backlogged mail.
3/2/97: The company that provides our Internet Connection will be running a test on our T-1 line that will take between 10-30 minutes. They will begin the test sometime between 12:30 p.m. and 1:30 p.m.. During this time you will be able to connect to us, check e-mail, look at local web pages (on our server), and visit newsgroups. You will not get a response if you try to use the world wide web for sites off our server.
1/31/97: We will have the network down for about 1 hour beginning at approximately 2:00 p.m. to update software. (We have determined that this time of day has the minimum number of users, we don't want to cause an interruption during high demand time). Update: We brought network down at 2:48 p.m. and came back up at 3:40 p.m.- I appears our updates are working correctly. Shortly after we came back online, we noticed our Windows 3.1 users could not authenticate and logon. We checked out the system and made the necessary changes and Windows 3.1 users could logon around 6 p.m.- sorry for the inconvenience. We will have to make routine upgrades to the hardware & software. We'll post when we have to do this again. Thanks.
1/14/97: Additional phone lines will be up and running by Friday, January 17th. We thought we would have these
lines up by the end of last week but they should be in place by the end of this week. Sorry for any logon problems caused by busy signals. We strive to have the necessary lines in place prior to demand exceeding supply. Update: Additional lines are in place and operating correctly.
1/2/97: MCI backbone network was down for about 20 minutes. They are a major backbone for Internet and caused a loss of service on rr1.net for the same period. You could connect to our machine but could not navigate. It affected about 30 million others at the same time so you had company.
12/28/96: The system was down from 10:55 p.m. to just before 1:00 a.m.- We will have a scheduled
maintenance in the next 72 hours to update some server software to correct this unscheduled downtime.
12/27/96: One of our phone lines in our hunt group is not routed properly by GTE. This will result in a ringing and
no answer by the network when this line is hit in the hunt group. We should have this corrected on Monday 12/30/96. UPDATE: The line is updated. It was not GTE it was a bad wire in the patch between the modem and
the serial card.
If you have any questions, comments, etc. e-mail firstname.lastname@example.org.