By Greg Stiles

For the Tidings

Scores of Southern Oregon business and government customers lost telephone service for several hours Thursday afternoon, reducing communications to cell phones and insufferably slow Internet links.

A device connecting several Southern Oregon telecommunications providers to points north operated by LS (Lightspeed) Networks of Portland failed shortly before 1 p.m. A backup switch for the transport node — which routes fiber traffic between points — was engaged, but the accompanying software was corrupted, triggering the failure, said local service providers affected by the malfunction.

One of those providers, Hunter Communications, dispatched technicians to the Qwest central office in Medford to make repairs for LS. Service was incrementally restored beginning shortly before 5 p.m., said Richard Ryan, Hunter Communications president.

"It's instantaneous when the software has a problem," Ryan said. "We were already working on a solution for this, but we weren't anticipating it would ever occur. They have built-in redundancy, so technically it should have never happened. Our technicians came up with a work-around solution that Lightspeed could remotely program."

Thousands of Internet users throughout the region experienced slowdowns during the failure.

Ryan said Internet activity wasn't totally lost because of safeguards built into the system.

The crash affected a variety of telephone, Internet and data providers.

The companies included Hunter, Ashland Fiber Network, Charter Communications, Clearwire, Infostructure, Integra and Rio Communications. Ryan said Southern Oregon Educational Service District and Southern Oregon University were hampered as well.

One by one, customers were brought back online as service to providers was restored. The failure couldn't have come at a worse time for some telephone customers.

"It knocked out our fax machine," said Cindy Bales, the controller at specialty painting contractor FD Thomas in Central Point, which competes for jobs all over the country.

"Most of our bids are sent out Wednesdays and Thursdays, so this was a bad time for the phones to go out," Bales said. "If we don't get our bids out, we're out of the running. If we don't get bids, we don't get work — that's our livelihood."

Bales said employees were able to shift internal communications to scattered work sites by cell phone or e-mail.

Slothful Internet service reduced activity at Ashland bio-med software firm Tree Star to a crawl.

"Our Internet connections were real spotty, hit and miss," said Tree Star systems administrator Ryan Minihan, whose firm is on the Ashland Fiber Network. "As people were returning from lunch they noticed there was trouble and it just got slower and slower."

Tree Star has 15 employees and shares a building with about 10 other workers, who depend on high-speed data transmission.

"It was a like a ghost town, all of a sudden," Minihan said. "We're pretty flexible here and people just went home."

Many of Tree Star's customers are on the East Coast, so afternoons tend to be slower. Had the slowdown rolled into today, however, it would have created more serious problems.

"It would be a real problem," Minihan admitted.

What was a headache for some firms was simply an inconvenience for others.

Ronell Matthews, who works at American Family Insurance's district office in Medford, said her staff could neither answer telephone calls nor retrieve voice mails.

"When I tried to make a telephone call at 12:30, it was something like just dead," Matthews said. "It was more of a hiccup for us, because we don't deal with customers quite so much as our other offices."

Calls to LS Networks for comment were not immediately returned and nothing pertaining to the incident was posted on the company's Web site Thursday afternoon.

John Nee, vice president of communications for Integra Telecom, said about 200 of its customers in the Medford area lost service.

Like other local providers, Nee said he was concerned that the redundancy factor failed.

Jeff Rhoden of Infostructure said some good came out of the failure.

"We now have another (fiber) route established in case we have another issued like this again," Rhoden said. "It took us about as much time to do that as it did to fix it."

Reach reporter Greg Stiles at 776-4463 or e-mail business@mailtribune.com.