In the end, all it took was a small chemical spill -- perhaps less than a barrelful -- to bring down the newest, mightiest oil refinery in the United States.

Three weeks ago, while workers repaired a minor leak at the Port Arthur, Texas plant owned by Motiva Enterprises, a few gallons a day of so-called "caustic" was inadvertently seeping into the newly built crude distillation unit (CDU), the 30-story-high network of interconnected cylinders and latticed pipelines at the heart of the refining process.

While harmless when mixed with crude, the undiluted caustic vaporized into an invisible but devastating agent of corrosion as the chamber heated up to 700 degrees Fahrenheit (370 Celsius); the chemical gas raced through key units, fouled huge heaters and corroded thousands of feet of stainless steel pipe.

Now, just weeks after they commissioned the biggest U.S. refinery project in a decade, two of the world's biggest oil titans -- Royal Dutch Shell and Saudi Aramco , which own Motiva -- are rushing to repair the potentially billion-dollar glitch that has added an embarrassing and costly coda to a landmark $10 billion expansion.

After a five-year effort to double the plant's capacity, making it the largest in the country, they must now reassemble many of the same people and parts for a blitzkrieg fix that may exceed the original $300 million cost of the unit: corrosion experts are flying in from across the world; hundreds of workers are being hired; bespoke 30-inch (75-c m) stain less steel pipelines and 30-story cranes may need to be obtained quickly, according to sources involved in the repairs.

Sources familiar with the effort provided Reuters with the most detailed account yet of what officials believe went wrong at the 325,000-barrels-per-day (bpd) unit known as vacuum pipestill-5 (VPS-5), showing how a series of seemingly minor glitches crippled the vast plant.

Too hot to handle

Motiva has said little about the incident. Late on Wednesday, 11 days after it occurred, the company confirmed for the first time that the unit might remain shut for "several months". Sources say officials are telling workers that the unit could be idle for as long as a year.

On Friday, in response to Reuters questions, Motiva spokeswoman Kayla Macke confirmed the contamination: "The preliminary inspection indicates that parts of the new unit have been contaminated with elevated levels of caustic."

The extent of the damage is still not known as portions of the crude unit are too hot to enter, according to the sources. Some areas may not be accessible for weeks.

Motiva has not reached a final conclusion as to the cause of the damage, but has developed a working theory on what experts said appeared to be a rare instance of "accelerated chemical corrosion". The unit's intense heat was critical: the rate of corrosion can double with every 10 degrees Celsius.

Even as it pitted the inside of the atmospheric section, a giant still that performs the initial and most basic stage of converting crude oil into fuel, the damage went undetected. Only when two fires broke out and a heater ruptured -- once crude resumed flowing -- did operators suspect something was amiss.

"They had the first fire and then they had the second one 20 feet away. They knew they had a problem," one of the sources said.

Why caustic continued flowing into the unit while it was idled to repair an unrelated leak is unclear, and is a key part of the investigation to establish cause. It is thought a valve failed to shut completely, but why that happened is unknown.

Meanwhile, every day the unit remains shut is an estimated $1.5 million in profit margin that Motiva isn't earning, and another 144 million miles (2 30 million km) wo rth of gasoline that isn't being supplied to the U.S. market during the height of the driving season.

As for oil prices, the incident will have a mixed effect. Unless it cuts production, Saudi Arabia will have to find new buyers for the crude that was earmarked for the refinery, weighing on oil prices. But the premium on regional wholesale gasoline may be pushed higher by the prolonged loss of supply, which should help buoy profit margins at rivals such as Valero and ExxonMobil.

Cooling down

While Motiva has been spared the kind of tragedy that struck BP's Texas City, Texas refinery in 2005, when 15 workers were killed and 180 injured in a vapor cloud explosion, the Port Arthur incident will likely stand as one of the most surprising to hit the U.S. refining industry in modern times.

The 4,600-acre (1,850-hectare) Port Arthur complex sits on a storied site. The first oil discovered in Texas was found 15 miles north of the plant in 1901; the state's first refinery was built on the Motiva site in 1903. A half-dozen other petrochemical complexes have given this Gulf Coast city a reputation as one of the most polluted in the country.

On May 31, with banners over a ceremonial industrial valve, Royal Dutch Shell Chief Executive Peter Voser and Aramco CEO Khalid Al-Falih celebrated the completion of a difficult project. The expansion began in 2005 but stalled in 2009 as markets tanked. It eventually ran $5 billion over the initial budget, and came in two years behind its original schedule.

"Rather than cut and run, we pressed ahead with our long-term commitment," Al-Falih said of Aramco's faith in the project, which he described as the company's largest investment outside Saudi Arabia. "We're confident the return on investment, despite the cost, will be very healthy."

With a total capacity to process 600,000 barrels of oil every day, the plant consumes more crude than the five-state, Rocky Mountain region. It can make 6 million gallons (23 million liters) of gasoline daily, enough to drive around the equator nearly six times.

By the end of May, the critical new units at the site had been running for about six weeks, enough time, officials thought, to address the usual teething pains and hiccups that often afflict massive new industrial complexes.

For example, the plant's new gasoline-and-diesel-making hydrocracker unit had reported flaring -- meaning it burned off hydrocarbon liquid and gas through a tower to relieve pressure in the system -- on May 28 because the wires for a temperature gauge were crossed, causing incorrect readings.

"You've got to be prepared for something to happen," said John Auers, senior vice president of Turner, Mason and Co, a Dallas refining consultancy. "There will always be issues with startup, and this was the biggest startup in U.S. history."

But only two days after the ceremony, on June 2, authorities decided to halt production temporarily to search for a leaking valve that was allowing vapor to escape from the crude unit, sources said. It was thought to be a minor problem.

It took nearly a week finally to stop the leak. Originally, it was thought a few bolts required tightening. Instead, clamps had to be set in place to secure the leak, the sources said. During that week the unit was kept out of production, but it wasn't completely shut down, which would have allowed the unit to go cold and taken longer to restore production.

Caustic invasion

While Motiva's VPS-5 was idling, authorities believe a few gallons each day of caustic leaked into the unit. The caustics are a base meant to negate the acid in cheaper heavy, sour crude that the new CDU was made to consume. They prevent residue from blocking pipes and reducing crude intake.

Normally, the amount leaked in the CDU would have been harmless, diluted by the crude. But only a small amount of hydrocarbon was circulating through the still while it was out of production, the normal method to maintain so-called "warm circulation" during a brief shutdown.

By the following weekend, unaware of the caustic incursion, Motiva began reheating the unit to resume operations; as the temperature reached 300 to 400 Fahrenheit, the caustic vaporized.

Ground zero was the atmospheric section, one of the simplest but most important machines in a modern plant. Although vast in scale, today's units are in many ways similar to the simple stills used to convert crude into kerosene for lamps at the start of the U.S. oil industry in the 1850s.

The core of any refinery, the main still boils crude at intense temperatures to split the hydrocarbon molecules into the initial components of fuels such as gasoline and diesel; the bulk of the output is an intermediate feedstock that requires further refining in a host of specialized secondary units.

Unlike a refinery blast, the misfortune unfolding at Motiva was relatively slow to materialize. The fires that erupted from small pipeline cracks that Saturday were small enough to be quickly extinguished by the workers on hand at the crude unit.

The extent of the damage was understood within two days.

"We have the worst-case scenario," one of the sources said. "Extensive damage throughout the crude unit. All of it."

Three engineering experts agreed that what one called "accelerated chemical corrosion" was rare, but not unheard of.

"The temperature issue could be a factor as well," said Kevin Garrity, president of NACE International, a global organization for engineers studying corrosion, and a 38-year veteran of the industry. He compared the effect to pouring sugar into hot tea, which dissolves the crystals much more quickly than in a cup of cold tea.

Normally, corrosion problems can be prevented.

"From a general sense you would not expect this kind of deterioration and problems in such a short period of time. You might not even expect it in 30 years if you have the right combination of technology and inspection practices," he said.

Repairs

Motiva has yet to examine the fractionation towers -- tall, thin, metal columns -- as well as the main part of the atmospheric section, because they are still cooling from their operating temperatures, said sources recruited for the repair work.

The vacuum section of the VPS-5 -- which takes the heaviest "residue" created in the atmospheric section and refines it in a vacuum, increasing the yield of feedstocks for other units -- was not damaged, they said.

Operators have continued to run many of the unharmed secondary units, although without the crude tanks they must buy intermediate feedstock from other refiners or shut peripheral units, as Motiva did last week.

But stainless steel piping, some sections as large as 30 inches in diameter, was damaged. Such equipment, part of more than 700 miles of pipe used on the expansion project, is often built to order, and may be difficult and costly to replace.

"If someone has 30-inch stainless steel pipe for sale, I would guess they're going to charge a premium price," one of the sources said.

Instrumentation on the unit is also known to be damaged, according to the sources.

Up to 50 heat exchangers will need to be cleaned throughout portions of the new plant, according to IIR Energy, an industrial intelligence firm that gathers data on operations and project activity on thousands of assets globally.

Work on exchangers 300 feet (90 meters) above the ground will require large cranes, though likely not the giants needed for the original construction. The heat exchangers, which look like 30-foot-long cylinders collected in a metal frame, house lengths of tubing where feedstocks are warmed and refined products are cooled as they go to and from a refining unit.

IIR also told Reuters that all trays in the distillation column and components within the furnace would need to be replaced. It said no restart timeframe had been determined.

Soon the question of who is to blame will arise. The cost to complete repairs may be as much as replacing the whole unit, which was originally estimated to cost some $300 million when the project was launched in April 2005, according to IIR.

Without knowing exactly why the caustic leaked, it's not possible to say who, if anyone, is at fault. The two main contractors for the project -- Bechtel and Jacobs Engineering -- declined to comment.

Meanwhile, the investigation continues, and oil traders await any word on when the plant will resume operations. Motiva will likely be "extra-cautious" in restarting, Auers of Turner, Mason and Co said.

"They're really focused on the repairs," one of the sources close to refinery operations said. "They don't need to know the cause now. They've got 12 months to figure that out and fix it."