1. Operations/Failure Analysis
Status of Equipment/Production Readiness
Plans in Case of Part/Systems Failure Per Stand

2. Inventory
We have just completed a complete inventory of all components of used in hybrid/module/rod testing
Thanks for all the help
The inventory is available at
In the process, we have identified many more potential failure modes
Lack of DAQ equipment, cables, Vienna box interlock spares, etc.
I have contacted all the people responsible for the components and hopefully we can receive all the spares we need before production fully starts

3. DAQ Components
The request for spare components has been out of over 5 months now. We still need:
2 TSC
1 FED (replacement for broken UCSB board)
3 TPO
2 eMUX cards
7 oMUX cards
2 CCU25
4 VUTRI
6 PAACB
11 hybrid-to-utri boards
All requests acknowledged and accepted. No ETA given for any of the parts

4. Why So Many #@*^%$ Cable Types?
We have 48 different cable types in the system
29 types have no spares
8 more have inadequate number of spares
Most can cause a complete system failures
I’ve requested spares for all cables that we can not make ourselves
Duccio has already responded
We can make a number of the cables ourselves
Especially ribbon cables for the TRHX box for the Module LT stands
We should coordinate the build

5. CAEN Power Supplies
With the UCR crate, we have the exact number we need
We’ve looked for spares and have yet to find any. Ariella said she would “steel” one for us.
Maybe Wiener could loan us their test stand in case of failure
Otherwise, the UCR stand will have to be used as a spare for the production sites
Keithley PS could be used in the short term
With the UCR ordered controller modules:
We will have spares of all modules at both UCSB and FNAL

6. Other Assorted Equipment Issues
UCSB needs to order/manufacture a spare of all components used in 4 hybrid test box
Coolie Box and Peltier spare status needs to be clarified
Both FNAL and UCSB need to find a spare NIM and VME crate
A spare set of LeCroy modules needs to be located for pulser logic of 4 hybrid thermal cycler
Spare computers with hybrid/module software needs to be made and tested for UCSB/FNAL
Spare computers with module LT/single rod/multi-rod/interlock software needs to made and tested for UCSB/FNAL

7. Test Stand Failure Analysis
I’ve made a first draft of the US testing operations/failure analysis document that Joe requested using everyone’s input so far
After checking the grammar, I’ll send it to the group for comments
All comments welcome. Hopefully we will figure out scenarios for all failures we can think of.
Process was really useful; got me to think of worse case failure scenarios

8. 4 Hybrid Thermal Cycler
Biggest strength is the large over-capacity we have in the group
We can test ~90 hybrids per day with expected peak rate of 45 hybrid per day assuming 400 per week from company
If global failure, move production to other sites
As long as we have a stockpile of bonded hybrids, it will not affect production. Otherwise, production slowed due to shipping times
The stands have three primary weakness: software, the Peltier element, and the NESLAB chiller
Backup computer and spare Peltier elements at each site reduces these risks
FNAL can borrow NESLAB chiller from BTeV. UCSB can purchase spare with 1-2 day lead time
UCSB obtaining spares of all other components which could be shipped overnight in case of failure

9. ARCS module testing We are in the best shape with these stands
We can test ~17 modules/stand/day with expected peak rate of production of 30.
FNAL has 4 stands, UCSB has 3 stands, UCR has 1 stand
Both sites have a complete live spare
Obtaining spare cables to reduce stand down-time to minimum
Repairs from Aachen have only taken 2-4 weeks in past
Biggest headache would be a complete failure of CMS database
No wire bonding data, sensor data, or hybrid data
Will have to find all fault during testing. May require each part to be tested twice.
Would have to check testing results against known failures after database is working. Running more than two stands should remove back-log of parts.

10. Module LT systems
Since expected production rate exceeds testing capacity (30 vs 15 modules a day) any failure would increase fraction of sampled modules
Only way to clear backlog is weekend testing
We are exploring reduction of tests to increase capacity to 20 per day
To prevent failures, spares acquired or ordered of almost all components:
Power supplies, DAQ components, cables, interlocks, etc.
Also modify Vienna box to be more stable/long-lived
Brass plates and extender connectors
Two major failures in which we don’t have spares in the foreseeable futures
CAEN crate, NESLAB chiller

11. Module LT systems (2)
In case of CAEN failure, we have to “borrow” UCR’s crate
In case of failed NESLAB chiller, 1-2 lead time for replacement
Maintenance really important
Without a chiller, I believe system could still be run without thermal cycling. The modules’ temperature should stabilize at C.
So no cold run information, but still “long term” testing
If the stand is complete non-operational, production can still continue
All production should be TOB. Only produce what can be assembled/tested on rod. Would reduce production capacity to ~20 modules a day

12. Single Rod System Test systems have over-capacity
2-4 rod assembled a day with ~8 rod test capacity/stand/day
Same issues with DAQ equipment/CAEN HV as module LT
Ordered 6 extra MUX to remove need of recabling
Cables from Duccio ordered: electrical and optical
Two pieces of equipment with no spares in foreseeable future: OEC & Delphi LV power supply
In both cases, take OEC or PS from multi-rod stand (with multi-rod loss of capacity of 7% and 12%)
If single rod stand fails completely, production can still continue a slower rate
Test rods as they are loaded into multi-rod stand. Adds 1 day per test cycle
Will have to reduce rod assembly rate to match testing rate
UCSB should switch to mostly TEC production

13. Multi-rod System Most complex system with least amount of experience
Same potential problems as module LT or single rod systems plus:
Chiller
Interlocks
Freezer infrastructure
UR have thought about different operational scenarios for these components; will have to be revisited after accumulation of more experience
Spares in hand of all components that company believe could likely fail
Compressor failure would lead to long down time
Plan for finding and removing leaks in the cooling system needed
1 or 2 spare C6F14 loads needed at both sites

14. Multi-rod System (2) Interlocks
Spares of sensors, etc. that can be easily replaced already ordered
If interlock hardware failed
Company has 48 hour express repair plan
Power supply interlocks would be used
Control of system by hand until repair made
If case of complete system failure, rod assembly at site can still proceed at a lower rate
All rods will have to be tested with single rod stand
Single rod stand would have to reproduce as many of the multi-rod stand’s testing until multi-rod available
Only way to remove the backlog of rods assembled would be the reduction of the testing cycle time.