JTAN News and Network Status

Mon, 04 Aug 2008

JTAN CA Certificate Re-Mint
Recently we were caught with our pants down by the 10-year expiration of our root CA certificate, first minted in 1998. Amazing. Who thought we would still be here, a decade later, actually still caring about x.509 certificates!? Why haven't they been obsoleted with something better? Why hasn't JTAN been obsoleted ... ummmm ... Better not go there.

Anyway, it was quite a fire drill, fixing this. We received scores of complaints about various secure services no longer working. I guess we have lots of customers who rely on these certs. Who knew? Fortunately, we were able to mint some new certs super-quick and get most everything working again.

Unfortunately, we were somewhat slapdash in our super-quick efforts. There were a few t's that we forgot to cross, and i's we forgot to dot when we minted our new CA certificate. Even though many services worked, some did not. Particularly members area certificate login, and VPN services.

In an effort to fix our CA correctly, we have re-minted all our certs. As before, this will cause some errors in browsers and email clients. Generally, clicking "OK" or "Accept" or "Make Exception" or "Add Certificate" will calm these errors. If the errors don't calm by simply saying "OK", you can download the new-and-improved JTAN CA certs here. Get your new client certs for VPN and passwordless members access through the members page.

Thank you for your patience and understanding as we figure this all out. With luck, we'll all be here 10 years in the future to do this all over yet again!

Posted Aug 04, 2008 at 17:54 UTC [/] permanent link

Tue, 22 Jul 2008

Downloading the New Certs
We have updated the links to download the new certs, as well as the MD5 and PGP signatures. You can get the JTAN CA root cert and the POP and SMTP certs here.

Posted Jul 22, 2008 at 02:58 UTC [/] permanent link

New JTAN CA Certificates
The JTAN Root CA certificate has been updated. We have also re-signed all the client certs, and certs for the POP/IMAP servers and the VPN. At least we think we have. Let us know via an email to tech-support if anything isn't working.

I need to point out that the old JTAN CA certificate was minted in 1998. While in one sense it's inexcusable that we forgot about its renewal in 2008, I think it's at least understandable that this slipped our minds. I mean, in 1998 I myself never dreamed that JTAN would exist in 2008, let alone that the cert would need renewal.

In any case, thank you for you understanding. You can get the new CA cert at our Certificate Authority site. You can download a fresh client cert from the members page.

Posted Jul 22, 2008 at 00:50 UTC [/] permanent link

Mon, 21 Jul 2008

JTAN CA Certificate Expired
The JTAN CA Certificate expired today. A new cert will be up shortly and you can update your certificate stores when the new cert is ready. We apologize for any inconvenience this may cause.

Posted Jul 21, 2008 at 21:44 UTC [/] permanent link

Thu, 03 Jul 2008

Fake Email Pretending to be From JTAN
We've heard several reports of emails that say they come from THE JTAN UP-GRADE SUPPORT TEAM that ask for your password. These emails are obvious fakes; I hope no one is fooled by them. All official mail from JTAN has the prefix {JTAN} in the subject line, and JTAN never, ever asks anybody to email passwords.

As a general rule, any email that asks you to supply a password or any personal information, either directly or indirectly, should be assumed to be a scam.

I repeat, all official mail from JTAN has the curly-brace prefix {JTAN} in the subject line, and JTAN never, ever asks anybody to email passwords.

Posted Jul 03, 2008 at 02:11 UTC [/] permanent link

Fri, 06 Jun 2008

Order Processing Suspended
Processing of routine orders for new features and installations that involve manual review or processing will be temporarily suspended for a week begining 1600 UTC today and ending 14 June. Orders for mail aliases, DNS changes, FTP and POP accounts will still be processed. Orders for new web sites, domains, servers, and any special orders will be delayed till the 14th.

We apologize for this interruption in service.

Posted Jun 06, 2008 at 12:47 UTC [/] permanent link

Mon, 02 Jun 2008

More News About the EV1 Disaster
This comes straight from their CEO

Dear Valued Customers:

... I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.

As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.

There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running. We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system.

During the upcoming week, we will be working with those customers to resolve issues. We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do...

Douglas J. Erwin
Chairman & Chief Executive Officer

Posted Jun 02, 2008 at 14:07 UTC [/] permanent link

Problems Continue After Explosion
EV1 has thus far failed to restore power and JTAN's accelerator servers are still not back online. We continue to expect restored service Real Soon Now.

Because of the EV1 outage in Texas, we have been forced to route some of the heavy traffic normally handled by the accelerators back onto our main servers in Conshohocken. This has caused a few delays, particularly in email. I apologize for the inconvenience. The initial burst of traffic seems to have settled, and things seem to be flowing smoothly again.

Although we expect accelerator service restored momentarily, if anyone decides they don't want to wait, we can swing your traffic over to Conshohocken if necessary. Email tech-support for instructions if you want to take this step.

Posted Jun 02, 2008 at 13:43 UTC [/] permanent link

Sun, 01 Jun 2008

Explosion in Data Center
If you use the JTAN web accelerator, or have multi-home hosting of your high volume site, this announcement is relevant to you.

Yesterday evening at 21:55 in the H1 data center of EV1 in Texas, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding the electrical equipment room Thankfully, no one was injured.

The JTAN web accelerator servers are hosted in this data center. They were not damaged or lost, but are currently without power.

EV1 administration staff has just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. EV1 was not allowed to activate their backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of the EV1 support team are in, and all vendors who supply them with data center equipment are on site. Their initial assessment, although early, points to being able to have some service restored by about 2100 UTC today.

Based on that ETTR assessment from EV1, JTAN will take no immediate action to re-engineer our accelerator. We expect everything to recover when they get the power back on.

We apologize for the outage this causes. Although no immediate action seems warranted, this event does illustrate a weakness in the design of our accelerator that we will attempt to fix in the coming weeks.

Thank you for your business.

Posted Jun 01, 2008 at 17:43 UTC [/] permanent link

Tue, 15 Apr 2008

New JTAN PGP Key
We have renewed our GPG/PGP signing key for JTAN customers. Anyone that wants their personal keys sign by us should submit them through the members page. We consider a signed key as equivalent to a members password or other strong proof that you really have an account at JTAN. Therefore, it's a good way to back up your password letter.

For more info, see our PGP page.

Posted Apr 15, 2008 at 14:03 UTC [/] permanent link

Fri, 28 Mar 2008

Mail Sending Wierdness
We have been getting some reports of problems sending mail, particularly with attachments. We aren't really sure what this problem is -- or even if it really is a problem. We have received other conflicting reports that mail is flowing OK.

It's possible that the network disruption on Monday caused some mails to be backlogged. We are investigating this possibility, but thus far we haven't found any logjams or errors to fix.

If anyone has a specific error message dated today, or later, that we can investigate, please pass it on to tech-support at jtan.com -- C

Posted Mar 28, 2008 at 16:14 UTC [/] permanent link

Tue, 25 Mar 2008

DDOS Attack
Begining at around 0600 UTC, and continuing currently, JTAN has been adversely affected by an ongoing DDOS attack. This attack has disrupted various services. We are attempting to deflect or ameliorate the ill effects of this attack and we believe that at this time most of our services are functioning despite the attack. If you notice a service that is broken, please let us know so that we can relocate it or tune it to avoid the problems.

Obviously, if anyone knows the miscreants responsible for this nonsense, please tell them to cut it out.

Posted Mar 25, 2008 at 15:01 UTC [/] permanent link

Thu, 14 Feb 2008

Network Disruption on .6 Network
It appears that there is some sort of network disruption that is advesely affecting addresses in the 207.106.6.X range. It's unclear what the cause of this is, or if it is even within JTAN, but we are aware of the problem and are investigating.

Posted Feb 14, 2008 at 12:02 UTC [/] permanent link

Fri, 11 Jan 2008

Thebe Out of Memory
The problem with thebe seems to be that it runs out of memory. Since there is plenty of memory on thebe normally, we can only conclude that some runaway program or web site is crashing, using it all up, and causing this issue.

We are trying to catch the errant process in the act of allocating all of memory, but thus far we have been unable to spot the culprit before the machine becomes inaccessible.

Anyone running something new on thebe (new relative to the last couple months) should be concerned that your new program, website, perl or php script, is the cause of this problem. It would help immensely if everyone on thebe would monitor the memory usage of their stuff and be on the lookout for the runaway.

Of course, those of you conscientious enough to monitor your stuff are probably not the people who would run something unstable in the first place. No doubt, the person running this has no idea that it's causing a problem and isn't even reading this notice.

In any case, we will eventually find the cause of these memory exhaustions and put an end to them. Thank you for your patience and understanding. This sort of thing in an inevitable disadvantage of shared hosting. If you have a web site where uptime is critical, consider using a JTAN dedicated server, starting at $79/mo.

Posted Jan 11, 2008 at 13:24 UTC [/] permanent link

Tue, 08 Jan 2008

Not So Fast (a new thebe issue)
When it rains, it pours. We are seeing a very high load average on thebe, making the machine very sluggish. We are investigating the cause of this.

If your web site is hosted on thebe, you may see some delays. Thank you for your patience as we investigate.

Posted Jan 08, 2008 at 21:25 UTC [/] permanent link

Mon, 07 Jan 2008

Thebe Work
Since thebe has exhibited similar flakey behaviour, we've decided to run RAM diagnostics on that machine as well. Therefore, if you have a website hosted on thebe, you will see some sporatic outages today as we verify the RAM.

Thanks again for your patience. Although we cannot stop our machines from breaking, we can work hard to fix them should they need it. Rest assured that we are working hard right now to resolve these issues.

Posted Jan 07, 2008 at 17:31 UTC [/] permanent link

Jupiter Fixed (again)
This time we think we really fixed jupiter. The problem that at first seemed to be a bad disk now appears actually to have been some bad RAM. We saw errors on comprehensive RAM diagnostics and replaced the faulty chips. Now the machine is checking out OK. We hope that this finally fixes the flakiness we've seen over the past few weeks.

Posted Jan 07, 2008 at 16:46 UTC [/] permanent link

Jupiter Down
Work continues on the repair of jupiter. Anyone with a web site on jupiter is experiencing an outage. We have a new disk being restored from backups, and we are doing RAM tests. These do take some time (many hours), but are necessary if we want to be sure we have correctly identified and fixed the problem. If our work goes well, we should have the machine back online by this afternoon.

Thank you for your patience.

Posted Jan 07, 2008 at 14:59 UTC [/] permanent link

Sun, 06 Jan 2008

Jupiter Issues
Jupiter is having issues with the web server. If your site is hosted on Jupiter, it may be inaccessible right now. We are working on fixing this problem. We appreciate your patience.

Posted Jan 06, 2008 at 14:00 UTC [/] permanent link

Mon, 24 Dec 2007

Happy Holidays

                            |                         _...._
                         \  _  /                    .::o:::::.
                          (\o/)                    .:::""":o:.
                      ---  / \  ---                :o:_    _:::
                           >*<                     ':}_>()<_{:'
                          >0<@<                 @    ''//\\''    @
                         >>>@<<*              @ #     //  \\     # @
                        >@>*<0<<<           __#_#____/'____'\____#_#__
                       >*>>@<<<@<<         [__________________________]
                      >@>>0<<<*<<@<         |=_- .-/\ /\ /\ /\--. =_-|
                     >*>>0<<@<<<@<<<        |-_= | \ \\ \\ \\ \ |-_=-|
                    >@>>*<<@<>*<<0<*<       |_=-=| / // // // / |_=-_|
      \*/          >0>>*<<@<>0><<*<@<<      |=_- |'-''-''-''-'  |=_=-|
  ___\\U//___     >*>>@><0<<*>>@><*<0<<     | =_-| o          o |_==_|
  |\\ | | \\|    >@>>0<*<<0>>@<<0<<<*<@<    |=_- | !     (    ! |=-_=|
  | \\| | _(UU)_ >((*))_>0><*<0><@<<<0<*<  _|-,-=| !    ).    ! |-_-=|_
  |\ \| || / //||.*.*.*.|>>@<<*<<@>><0<<@>>>>>|  ( ~~~ )/   (((((((())))))))
      ~~~~~~~~         '""""'------'  'w---w'     '------------'

                          Happy Holidays from JTAN

Posted Dec 24, 2007 at 19:27 UTC [/] permanent link

Sun, 23 Dec 2007

Jupiter and Thebe Issues Seem Resolved
Jupiter is back up; thebe has been back up for a few days. We aren't quite sure why these two machines started acting up recently, other than the general superstition that machines will break on holiday weekends so that overworked system administrators can have even more reason to complain about their thankless jobs.

In any case, we hope these machines are happy now. They seem to be working OK. Thanks for the reports, and your patience.

Posted Dec 23, 2007 at 19:40 UTC [/] permanent link

Jupiter Webserver Issue
There now seems to be some issue with the jupiter web server. If your site is hosted on this machine, it is currently down. We are investigating the problem.

Posted Dec 23, 2007 at 14:19 UTC [/] permanent link

Wed, 12 Dec 2007

Thebe Issues
There seems to be some issue on the the machine thebe.jtan.com. If your web site is hosted on this machine, your site is currently down. We are investigating the problem. Thank you for your patience.

Posted Dec 12, 2007 at 01:06 UTC [/] permanent link

Sat, 03 Nov 2007

Anti-Spam Defenses Dropped?
Since our new strict IP/hostname policy has been so effective in blocking spam, we have dropped several techniques that we previously used for blocking spam. The most significant of the techniques dropped is the "LATE_FILTER" system based on our realtime spammer database.

We probably spent nearly $50,000 creating that LATE_FILTER system, which rejected many of the same spammers that our new, simpler, more effective system rejects. Despite how much it hurts me to decommission a system that virtually everyone at JTAN has worked on for the last 3 years, there's no question that the new system works better.

We are keeping the old system around in the form of a header tag. You now may occasionally see the header "X-JTAN-JRBL: yes" added to a mail. This is an indication that the old system would have blocked the mail. For those of you that want super-fascist blocking, you can add a rule that uses the above header. Keep in mind, however, that we no longer support that header.

Anyway, I think it's fabulous that we can be eliminating anti-spam systems, no matter how much money we wasted on them. I want these spam wars to end, and this may be the first indication that an end is in sight.

Posted Nov 03, 2007 at 15:24 UTC [/] permanent link

Wed, 31 Oct 2007

Timezone Change
JTAN will be migrating all its public servers and mail machines to a default timezone setting of UTC. We have customers all around the globe and this seems the best choice for everyone.

In the past we had set many of our machines to default to EST5EDT, but the recent legislation in the USA that changed when clocks are supposed to switch from summer time to standard time has made this an unattractive default.

Shell users that want to customize their timezone to their own location could always do so using the TZ environment variable as explained in tzset(3) and environ(7). This is still possible, of course.

Basically, the TZ variable controls the timezone to use when displaying dates. The normal format is a pathname relative to /usr/share/zoneinfo. For example, setting TZ=US/Eastern displays the current time in the US east coast states according to the new law.

Posted Oct 31, 2007 at 01:34 UTC [/] permanent link

Mailman Fixed
The issue with Mailman lists has been corrected.

Posted Oct 31, 2007 at 01:14 UTC [/] permanent link

Tue, 30 Oct 2007

Mailman Issue
There seems to be an issue with mailman mailing lists. This is unrelated to the new IP policy. We have been receiving increasing reports of web errors and missing list mail. The problem is under investigation.

Posted Oct 30, 2007 at 02:25 UTC [/] permanent link

Mon, 29 Oct 2007

Dramatic Decrease in Spam
Because of our new, stricter IP policy, there has been a dramatic decrease in the amount of spam entering JTAN mailboxes. The improvement is incredible. Many customers have already complimented the new policy as having virtually eliminated spam from their INBOX.

That's the good news. The bad news is that we have blocked mail reports from a very tiny minority of legit senders that have set up their mail servers on cable modems and DSL lines. These senders choose not to configure their hostname and not to use the SMTP server offered by their provider, and therefore no longer can send us mail.

Thus we are faced with a difficult choice. On one hand, we have the vast majority of our customers who demand reliable, spam free email service. On the other hand, we have a few senders who for some reason cannot configure the hostname of their email server, nor can they use a recognized relay, yet still demand to have their email delivered to JTAN just the same.

We are having trouble understanding the argument of the blocked senders. From our perspective, on the Internet today it is clearly due diligence for an email sender to have a proper text hostname. We are not telling anyone what their hostname must be, just that it must not look the same as the millions of numeric hostnames that are sending us spam from dynamic DSL and Cable addresses. We expect that someday all the cable and DSL providers will block port 25 (as many already have). On that happy day, this point will be moot.

Of course, we are willing to listen to special cases. We can make special arrangements for any legit sender that is prohibited the use of a proper hostname to send us mail, and also cannot use their own ISP's relay. But we will do this only after the sender has shown that they have exercised due diligence in attempting to implement these other two options. JTAN has expended large amounts of resources (time, money, and lives) in the battle against spam. We do this for our customer's benefit, not to benefit senders that won't exercise commonly recognized due diligence in configuring their mail systems.

Possibly the worst case we may face is a valued JTAN customer who has a correspondent that is unwilling to change. The JTAN customer demands that we whitelist his unbending correspondent. I'd hate to be forced into making that choice. We would hope that all senders that find themselves blocked by this stricter policy will be all too happy to join us in the fight against spam and make the small changes needed so as to be able to send us mail again. And we also hope that JTAN customers that continue to demand reliable email service understand that JTAN can't do this without cooperation from legit senders.

Posted Oct 29, 2007 at 23:08 UTC [/] permanent link

Sun, 28 Oct 2007

Stricter IP Address Policy on Incoming Mail
We have begun to implement policy that will reject mail messages that are sent to us directly from dynamic IP addresses or IP addresses that have missing or mal-formed reverse DNS names. From what we can tell, this change will cut spam dramatically and have no negative effect on legit senders.

If, however, this policy somehow gores a legit sender, we will do what we can to help.

Posted Oct 28, 2007 at 21:24 UTC [/] permanent link

Mon, 10 Sep 2007

Numerous Mail Issues Resolved
Our new RAID based POP/IMAP server seems to be operating quite well. We have received many reports that webmail is far faster now. And of course, knock on wood, the stability of the new system has been solid thus far.

A fortuitous consequence of bringing up the new POP/IMAP server was that it led to solving a few nagging puzzles, particularly an annoying issue with outgoing mail. For a long time we had been receiving complaints about outgoing mail failing to certain ISPs when the message length was above a certain minimum. The bounce message said "I/O Error" or "No Route to Host". We believe we now know the cause and workaround for this subtle issue -- an issue that isn't exactly JTAN's fault, but one we must address none the less. It was through our POP/IMAP upgrade that we cracked this puzzle. Sometimes luck is with us.

In any case, I'd like to thank you all one last time for bearing with us through the disaster that filled the last few weeks. As always, we struggle to keep our mail system up to date and fully operational. We understand the importance of email.

Posted Sep 10, 2007 at 19:20 UTC [/] permanent link

Sun, 09 Sep 2007

Local Mail Delay
We are getting reports that local mail (JTAN-to-JTAN) is delayed. It's possible this is a consequence of the POP/IMAP replacement. We are investigating.

Posted Sep 09, 2007 at 21:40 UTC [/] permanent link

Fri, 07 Sep 2007

Re-subscribing to Mail Folders
A number of users have reported that their IMAP folders have disappeared. Never fear, they should still be there. What you need to do is re-subscribe to your folders. In Squirrel Mail you will find this option under the "Folders" screen. Other mailers will have a similar button for re-subscribing.

We have also heard reports that the "seen" flag on messages has been reset, and that some mail has been duplicated. I'm sorry to confirm that these seem to be unfortunate artifacts of the move. The whole mail database, many Gigs in size, was recreated afresh. All we could preserve was the mail itself.

N.B. JTAN has always recommended that users do not rely on JTAN mail folders for long term storage of their mail. Although we do everything we see as reasonable to protect this data, we do not guarantee there won't be data loss.

Posted Sep 07, 2007 at 21:25 UTC [/] permanent link

Transition To New POP/IMAP Server Completed (we think)...
For everyone that has been suffering from our flaky POP/IMAP services in the last few weeks, relief may be in sight.

We believe that we have successfully transitioned to a new POP/IMAP server, our doubtful precognitive abilities not withstanding. If there is any order in this universe, the total upgrade of the POP/IMAP system will fix the instability we have been seeing in the old system.

The new system is still based on Cyrus IMAPd from Carnegie Mellon, but now uses the latest version. Also, all the email boxes are now on a RAID system. We are thinking this will provide better availability than what we saw with the previous system that did not use RAID, at the sacrifice of some non-critical reliability.

We definitely know how important email is to everyone. Our phone has been ringing off the hook with complaints. In those brief periods when email actually has been working, our tech-support mailbox has been filling with complaints as well. I'd like to thank you all for your patience with this. I know it has been difficult.

No doubt there are some issues we missed with the new server. We will are rebooting it now one final time for good luck (at about 16:45Z). Afterward, please report any issues you may see.

Thanks again for your business.

Posted Sep 07, 2007 at 16:44 UTC [/] permanent link

POP/IMAP Emergency Maintenance
We are actively working on the POP/IMAP server at this time. Availability will be spotty today, but we hope this will be the last major outage.

Please understand that we feel your pain tenfold when email is down. While we did forsee this disaster developing, and did take steps to have a replacement system available, the old system deteriorated faster than we anticpated. Thus, we have been unable to do this transition without downtime.

Posted Sep 07, 2007 at 13:33 UTC [/] permanent link

Wed, 05 Sep 2007

POP System Upgrade
The recent instability with the POP/IMAP system has convinced us that the current system needs to be replaced. We are preparing an upgrade. That's the good news. Unfortunately, given the immense quantity of email people store in their mailboxes (up to a GIG per mailbox), moving from the old system to the new system is not easy to do. Simply moving the mail from the old disk to the new system will take a long time.

Thus, you can expect for the next few days we will be moving to the new system. We will try to do this with minimum downtime, but it will be inevitable that POP service will be interrupted from time to time. In addition, old mail may temporarily disappear. New mail may appear twice.

I expect that the whole process will take about two weeks. If we are successful, the stability of the POP/IMAP system should improve. Thank you for your patience and understanding.

Posted Sep 05, 2007 at 21:46 UTC [/] permanent link

POP Server Outage
It appears that the POP server is acting up again. We will do some emergency maintenance on it in the next couple hours in an attempt to fix the problem. Stay tuned.

Posted Sep 05, 2007 at 16:55 UTC [/] permanent link

Thu, 26 Jul 2007

Bay Area Power Outage -- UPDATE
JTAN continues to be up and running just fine, but yesterday several of our customers located in the San Francisco area were seeing spotty Internet service. We believed the problem was related to a datacenter power outage in their city as described here. We still believe this was the ultimate cause, but we now know more detail.

As it turned out, the key issue was at Level 3, one of those giant dot-coms. Apparently even these giants can have outages. Their transcontinental route was broken, and California fell into the sea (in a metaphorical internet sense). Of course, since most of the major internet sites are hosted in California, while the rest of the country lost major sites like Netflix and Craigslist, from the point of view of the poor folks in San Rafael, all they lost was JTAN. Hardly a fair trade for poor Marin County.

Anyway, I'm told that the hard working techs at Level 3 have their transcontinental route fixed now. If anyone in San Rafael is still having trouble connecting outside of CA, that is, to JTAN, please let us know and we'll attempt to find out more if we can.

Again, JTAN remains just fine. None of our equipment was affected in the slightest. None of the services we provide to the rest of the world were interrupted by this outage at Level 3.

Posted Jul 26, 2007 at 11:14 UTC [/] permanent link

Tue, 17 Jul 2007

Network Outage
From approximately 2100 till 2300 UTC the JTAN network was disrupted by a DOS attack.

Posted Jul 17, 2007 at 00:32 UTC [/] permanent link

Wed, 11 Jul 2007

POP/IMAP Server Emergency Maintenance
Many of you have reported errors when checking email. We have also noticed some instability in the main POP/IMAP server. These seem to occur most in the early morning (relative to UTC-4) and for that reason, initially, we thought the problem was a slowdown during backup. Now it's clear that there's more of an issue.

Tomorrow at approximately 1200 UTC we will do some quick maintenance on the POP/IMAP server for the purpose of investigating this problem. On or about that time you may see some errors when checking email.

Thank you for your patience and cooperation. We understand how important email is to you and we are performing this investigation so as to keep your email service running.

Posted Jul 11, 2007 at 23:32 UTC [/] permanent link

Sun, 08 Jul 2007

NOC Move: Done
Preliminary tests seem to indicate that our NOC move has been a complete success. I don't want to jinx it -- I'm sure we will find some big problems -- but thus far, the glitches have been small and performance in the new data center seems outstanding.

Please let us know by an email to tech-support if you see any issues.

Posted Jul 08, 2007 at 23:39 UTC [/] permanent link