1. Study of ATLAS Sensitivity to FCNC Top Quark Decays J.Carvalho, N.Castro, L.Chikovani, T.Djobava, J.Dodd, S.McGrath, A.Onofre, J.Parsons, F.Valoso Workshop, Tbilisi, October 18/2005 ATLAS

2. Workshop,Tbilisi,18/2005 Introduction We study the ATLAS experiment sensitivity to FCNC top quark decays ( t→Zc/u, t→γc/u, t→gc/u ) at the LHC In SM, FCNC top quark rare decays are strongly suppressed (loop suppression, heaviness of gauge bosons ) Br < 10 -13 - 10 -10 Supersymmetric extensions to the SM lead to significantly larger predictions for FCNC branching ratios on the level 10 -8 - 10 -6 still too small to be observable at the LHC Observation of significantly larger Br’s will be a clear signal of new physics ( new dynamical interactions of top quark, multi-Higgs doublets, exotic fermions or other possibilities ) The Supersymmetric models (without R-parity) have much higher value : Br ~ 10 -4 Although FCNC processes associated with the decay of top quarks ( t→γc/u, t→Zc/u ) have been studied at CDF, the amount of data collected up to now is not comparable with the statistics expected at LHC

3. Workshop,Tbilisi,18/2005 Introduction LHC will be a” top factory” NLO prediction that  (t-tbar) = 833 pb ( which is  100X larger than at FNAL) in pp collisions at √ s = 14 TeV implies that more than 8 thousands t-tbar pairs per day will be produced in low luminosity (10 -33 cm -2 s -1 ) running of LHC Production processes at LHC: Top Quark Pair Production through gluon fusion and quark-antiquark annihilation 90% gg + 10% q-qbar g g g t tt q qq t tt

4. Workshop,Tbilisi,18/2005 Introduction The Analysis presented here focus on two final state topologies:  Leptonic mode t → Xq ( X = Z, γ, g ; q = u,c ) t → Wb →lνb  Hadronic mode t → Xq ( X = Z, γ, g ; q = u,c ) Different types of analysis ( sequential & probabilistic ) are applied to each FCNC decay mode and their results are compared t → Wb →jjb

5. Workshop,Tbilisi,18/2005 Introduction The results obtained by the different analyses is presented within two approaches : Branching ratio sensitivities ( assuming a 5σ signal significance for discovery ) 95% confidence level limits ( in the hypothesis of signal absence)

6. Workshop,Tbilisi,18/2005 Monte Carlo Event Generation The MC generation of the background processes : b-bbar, Z/γ* * +jets, WW, ZZ, ZW } PYTHIA. W+jets } HERWIG Single top quark production, SM t-tbar & Signal } ONETOP & TopReX Initial and final state QED and QCD radiation (ISR+FSR), multiple interactions, fragmentations and decays of unstable particles were enabled. The generated background and signal events were passed through the ATLAS fast simulation ATLFAST 2.0, 2.14 or 2.53 and ATLFAST-B packages. ATLFAST was used with its default parameters for low and high luminosity running. The b-jet tagging efficiency was set to 60% for low luminosity ( L= 10 fb -1 ) and 50% for high luminosity ( L = 100 fb -1 ) The rejection factor of c - jets is 10 and of light quarks jets 100. The CTEQ2L and CTEQ5L Parton Distribution Functions were used.

7. Workshop,Tbilisi,18/2005 t → Zq ( q = u, c ) CDF Br < 33% @ 95%CL The enormous QCD backgrounds at hadron colliders make the search for the signal via hadronic decays t-tbar → WbZq → 6-jets very difficult. Z → l + l - ( l = e,m ) significantly reduces the backgrounds provides a relatively clean experimental signature. The final state is determined by the decay of W: * t-tbar → Zq,Wb → llj, l νb leptonic mode * t-tbar → Zq,Wb →llj, jjb hadronic mode The hadronic W decay mode has a much larger Branching ratio, but suffers from larger backgrounds.

8. Workshop,Tbilisi,18/2005 t-tbar → Zq,Wb → llj, l νb Sequential Analysis ( leptonic mode ) Experimental Signature: ● three i solated charged leptons ( with m_ ll ≈ m_ z ) ● missing transverse momentum p T ● two jets : light _jet + b_jet Backgrounds : * Z+jets dominated by ● qg→ Zq ; ● qqbar→Zq thresholds were imposed at the generator level on the invariant mass, m^=√s=130 GeV and transverse momentum ( p T > 50 GeV )hard parton interaction. σ = 3186 pb 2.1 X 10 7 events * ZW ( 26.58 pb ) ; * Wjj ( 18000 pb ) * Wbb ( 300 pb ) * Single top: W-gluon Fusion (244 pb ), Wt (60 pb ), s-channel (10 pb ) * t-tbar SM ( 833 pb ) : dileptonic decay, semileptonic decay, hadronic decay

9. Workshop,Tbilisi,18/2005 t-tbar → Zq,Wb → llj, l νb Sequential Analysis ( leptonic mode ) The final topologies of most dangerous backgrounds: ● WZ→ l + l - l ± ν 38 000 events ● t-tbar→W + bW - b→l + νbl - νb 3.9 x 10 6 events selection cuts: ● three isolated carged leptons with p T l > 30 GeV (e), |η| < 2.5; Z+jets, t-tbar,Wjj,Wb-bbar,Wt ● at least two jets with p T jet > 50 GeV, |η| 0.4, ΔR lj > 0.4 ; ZW ● p T miss > 30 GeV Z+jets ● A like sign, same-flavor pair of isolated leptons was required to reconstruct to the Z mass within a window of m_ll= m z ± 6 GeV, Zmass resolution ~ 2.9 GeV. t-tbar ● A peak at the top quark mass in the Zj invariant mass distribution within mass window m - llj = m t ± 12 GeV, m - llj = m t ± 24 GeV ( The top quark mass resolution σ( m - llj ) = 14 GeV ) after all cuts, have (for L=100 fb-1) signal efficiency = 6.1 % number of surviving background events : 2 WZ, 4 t-tbar

10. Workshop,Tbilisi,18/2005 a)Invarint mass distribution of lepton pairs m-ll b)Invarint mass distribution of t→Zq→llj for leptonic mode

11. Workshop,Tbilisi,18/2005 t-tbar → Zq,Wb → llj, jjb Sequential Analysis ( hadronic mode ) Experimental Signature: ● two isolated charged leptons ( with m_ ll ≈ m_ z ) ● four jets : three light _jet + b_jet Backgrounds : * Z+jets dominated by ● qg→ Zq ; ● qqbar→Zq thresholds were imposed at the generator level on the invariant mass, m^=√s=130GeV and transverse momentum ( p T > 50 GeV ) hard parton interaction. σ(Z+jets) = 3186 pb 2.1 X 10 7 events * ZW ( 26.58 pb ) ; * Wjj ( 18000 pb ) * Wbb ( 300 pb ) * Single top: W-gluon Fusion (244 pb ), Wt (60 pb ), s-channel (10 pb ) * t-tbar SM ( 833 pb ) : dileptonic decay, semileptonic decay, hadronic decay

12. Workshop,Tbilisi,18/2005 t-tbar → Zq,Wb → llj, jjb Sequential Analysis ( hadronic mode ) The final topologies of most dangerous backgrounds: ● Z +jets→ l + l - + jets 2.1 x 10 6 events ● t-tbar→W + bW - b→l + νbl - νb + l ± νbjj 28.x 10 6 events selection cuts: ● two isolated charged leptons with p T l > 30 GeV (e), |η| < 2.5; t-tbar (semileptonic decay mode) ● at least four jets with p T jet > 50 GeV, |η| 0.4, ΔR lj > 0.4 ; WZ ● A like sign, same-flavor pair of isolated leptons was required to reconstruct to the Z mass within a window of m z ± 6 GeV, Z mass resolution ~ 2.9 GeV. ● invariant mass m jj = m w ± 16 GeV, W mass resolution ~ 8 GeV. Z+jets ● one jet tagged as b-jet Z+jets,WZ ● invariant mass m jjb = m t ± 8 GeV, m jjb mass resolution ~ 18.5 GeV. WZ & Z+jets ● A peak at the top quark mass in the Zj invariant mass distribution within mass window m - llj = m t ± 12 GeV, m - llj = m t ± 24 GeV, Z mass resolution σ( m - llj ) = 14 GeV after all cuts, have (for L=100 fb-1) signal efficiency = 0.4 % Number of surviving background events : two Z+jets, four t-tbar

13. Workshop,Tbilisi,18/2005 a)Invarint mass distribution of jet pairs, m-jj b)Invarint mass (m-jjb) distribution of t→Wb→jjb ( hadronic mode)

14. Workshop,Tbilisi,18/2005 a)Invarint mass distribution of lepton pairs m-ll b)Invarint mass(m-llj) distribution of t→Zq→llj ( hadronic mode )

15. Workshop,Tbilisi,18/2005 t-tbar → Zq,Wb → llj, l νb Probabilistic Analysis ( leptonic mode ) selection cuts: ● three isolated charged leptons with p T l > 25GeV (e), |η| < 2.5; Z+jets, t-tbar,Wjj,Wb-bbar,Wt ● at least two jets with p T jet > 20 GeV, |η| <2.5, ZW ● p T miss > 20 GeV Z+jets ● A like sign, same-flavor pair of isolated leptons was required to reconstruct to the Z mass by calculating m- ll invariant mass. t-tbar ● the top quark mass ( m - llj ) reconstruction was done by associating the non- b- jet with lepton pair. Last cut : ● L R > 5.62

16. Workshop,Tbilisi,18/2005 t-tbar → Zq,Wb → llj, l νb Probabilistic Analysis ( leptonic mode ) Following the final selection, a probabilistic type of analysis was applied. Signal ( Ρ i signal ) and background-like ( Ρ i back. ) probabilities were computed using Probability Density Functions ( p.d.f. ) constructed from relevant physical variables. The signal L s =Π i= 1 n Ρ i signal and background L Β = Π i= 1 n Ρ i back likelihoods (n is the number of p.d.f.) were used to built the discriminant variable, defined as L R = L s / L Β. For the t→Zq channel the p.d.f. were based on the following physical distributions : ■ minimum invariant mass m-ll of the three possible combinations of two leptons (only the three leading leptons were considered ); ■ transverse momentum of the third lepton p T l3 ■ the jll invariant mass and ■ transverse momentum of the most energetic non-b jet p Tjet

17. Workshop,Tbilisi,18/2005 Invarint mass (m-llj) distribution of t→Zq→llj Probabilistic Analysis ( leptonic mode )

18. Workshop,Tbilisi,18/2005 Distribution of variables based on wich the p.d.f were built (t→Zq) SM background and signal discriminant variable distributions

19. Workshop,Tbilisi,18/2005 t-tbar → γ q,Wb → γ j, l νb Sequential Analysis ( leptonic mode) ■ CDF Br < 3.2% @ 95%CL ■ The presence of a high p T photon in t-tbar → γ q,Wb events is not sufficient to reduce the QCD multy-jet background to a manageable level. Therefore, the analysis considered the final state with a the leptonic decay of W boson. Experimental Signature: ● one i solated high p T photon ● one i solated high p T charged lepton ● missing transverse momentum p T ● two jets : light _jet + b_jet Backgroundi s dominated by the events with a real W→lν and either a real or fake photon.: * Wjj +Wbb HERWIG * Single top: W-gluon Fusion,Wt (60 pb ), s-channel (10 pb ) ONETOP * t-tbar SM ( 833 pb ) : dileptonic decay, semileptonic decay, hadronic decay PYTHIA

20. Workshop,Tbilisi,18/2005 t-tbar → γ q,Wb → γ j, l νb Sequential Analysis ( leptonic mode) selection cuts: ● one isolated photons with p T γ > 40 GeV (e), |η| < 2.5; ● one isolated charged leptons with p T l > 20 GeV (e), |η| < 2.5; ● # jets = 2 with p T jet > 20 GeV, |η| 30 GeV) ● p T miss > 20 GeV ● invariant mass m(lνb) = m t ± 20 GeV ● A peak at the top quark mass in the m-γj invariant mass distribution within mass window m -γ j = m t ± 12 GeV, The top quark mass resolution σ( m -γ j ) = 7.7 GeV after all cuts, have (for L=100 fb-1) Number of surviving 155 background events dominated by t-tbar, Wjj,+Wb-bbar,Wt (60)

21. Workshop,Tbilisi,18/2005 t-tbar → γ q,Wb → γ j, l νb Probabilistic Analysis ( leptonic mode) selection cuts: ● one isolated photon with p T γ > 75 GeV, |η| < 2.5; ● a maximum of two isolated charged leptons with p T l > 20 GeV (e), |η| < 2.5; ● # jets = 2 with p T jet > 20 GeV, |η| 30 GeV) ● p T miss > 20 GeV ● A peak at the top quark mass in the m-γj invariant mass distribution Just like t→Zu(c) channel, a probabilistic type of analysis was used. The p.d.f. were built based on the following variables : ■ transverse momentum of the leading photon p T γ ■ the γj invariant mass ■ the number of jets Last cut : ● L R > 3.13

22. Workshop,Tbilisi,18/2005 The Distribution of variables based on which the p.d.f were built t → γ u(c) Probabilistic Analysis ( leptonic mode)

23. Workshop,Tbilisi,18/2005 SM background and signal discriminant variables distribution for t → γ u/c Probabilistic Analysis ( leptonic mode)

24. Workshop,Tbilisi,18/2005 t-tbar → gq,Wb → jj, l νb Probabilistic Analysis ( leptonic mode ) Experimental Signature: ● one i solated high p T charged lepton ● missing transverse momentum ( p T ) from the undetected neutrino ● at least three isolated jets : two light _jets+ b_jet Fully hadronic final state Backgrounds : * Z+jets * W + jets * Single top: W-gluon Fusion,Wt * t-tbar SM : dileptonic decay, semileptonic decay A search for a FCNC t-gluon-q coupling (with q=u,c) through the decay t→gq would be overwhelmed by background from QCD multi-jet events ■ More restrictive event selection is necessary

25. Workshop,Tbilisi,18/2005 t-tbar → gq,Wb → jj, l νb Probabilistic Analysis ( leptonic mode ) selection cuts: ● one isolated charged lepton with p T l > 20 GeV, |η| < 2.5; ● no photons with p T γ > 5 GeV (e), |η| < 2.5; to reject events assigned to t→γq ● at least three jets with p T jet > 20 GeV, |η| 40GeV) ● p T miss > 20 GeV The events were classified as ‘3 jets’ or ‘4 jets’ The ‘3 jets’ sample ■ transverse momentum of the leading jet p T g > 75 GeV ■ the gj invariant mass 125 <m_t < 200 GeV As for the other channels, a probabilistic type of analysis was adopted, using the following variables to built the p.d.f. : ■ the gq invariant mass ■ the blν invariant mass ■ transverse momentum of the b-jet ■ transverse momentum of the second non-b jet ; Last cut : ■ angle between the lepton and the gluon α l g ● L R > 1.13

26. Workshop,Tbilisi,18/2005 The Distribution of variables based on which the p.d.f were built t → g u(c) Probabilistic Analysis ( leptonic mode) ‘3 jets’

27. Workshop,Tbilisi,18/2005 SM background and signal discriminant variables distribution for t → g u/c Probabilistic Analysis ( leptonic mode) ‘3 jets’

28. Workshop,Tbilisi,18/2005 t-tbar → gq,Wb → jj, l νb Probabilistic Analysis ( leptonic mode ) The ‘4 jets’ sample ■ transverse momentum of the leading jet p T g > 100 GeV ■ the gj invariant mass 150 <m_t < 190 GeV The p.d.f. were built based on the following physical distributions : ■ the blν invariant mass, ■ transverse momentum of qg, ■ transverse momentum of blν, ■ angle between the lepton and the gluon ( α gl ), ■ angle between the lepton and b-jet ( α lb ), ■ angle between the gluon and the second non-b jet ( α gq ). Last cut: ● L R > 1.13

29. Workshop,Tbilisi,18/2005 The Distribution of variables based on which the p.d.f were built t → g u(c) Probabilistic Analysis ( leptonic mode) ‘4 jets’

30. Workshop,Tbilisi,18/2005 SM background and signal discriminant variables distribution for t → g u/c Probabilistic Analysis ( leptonic mode) ‘4 jets’

31. Workshop,Tbilisi,18/2005 Branching Ratio Sensitivity ( 5σ significance discovery hypothesis ) Expected top quark FCNC decay branching ratios sensitivities of the ATLAS experiment were estimated for both the sequential and probabilistic analysis under two different hypothesis: 95 % confidence level limits ( hypothesis of absence of signal ) 5σ significance discovery hypothesis Assuming a signal discovery with a 5σ significance, the branching ratio for each channel studied is estimated by: 5√B Min. BR = ---------------------------------------------- 2 x L x σ(t-tbar)x BR x ε

32. Workshop,Tbilisi,18/2005 Branching Ratio Sensitivity for a luminosity L =10 fb -1, L =100 fb -1 ( 5σ significance discovery hypothesis ) t→Zu (c) AnalysisLeptonic Mode Hadronic Mode Combined Result Sequential 4.70 x 10 -4 1.14 x 10 -4 1.70 x 10 -3 4.58 x 10 -4 4.50 x 10 -4 1.15 x 10 -4 Probabilistic5.10 x 10 -4 1.60 x 10 -4 ---- ----

33. Workshop,Tbilisi,18/2005 Branching Ratio Sensitivity for a luminosity L =10 fb -1, L =100 fb -1 ( 5σ significance discovery hypothesis ) Analysis t → γu(c) t → g u ( c ) 3jets 4jets combined Sequential 1.00 x 10 -4 --7.4 x 10 -3 Probabilistic 1.2 x 10 -4 3.8 x 10 -5 4.6 x 10 -3 1.4 x 10 -3 7.4 x 10 -3 2.3 x 10 -3 4.6 x 10 -3 1.4 x 10 -3

34. Workshop,Tbilisi,18/2005 95 % confidence level limits ( hypothesis of absence of signal ) Sequential Analysis For the sequential analysis of t  Zq decays at integrated luminosity 100 fb -1 was evaluated the 95% CL upper limits. Assuming the Poisson processes with backgrounds 95% CL upper limits on the number of signal events for both, leptonic and hadronic decay modes were derived. Using the NLO Calculation for σ(t-tbar), these limits were then converted into limits on the Branching Ratios for both decay modes. Probabilistic Analysis The modified frequent likelihood method was used to evaluate the 95% CL upper limits. the Full information of the discriminant variables were used to derive 95% CL upper limits on the number of signal events for each channel. No cuts on the discriminant variables were used. Using the NLO calculation for σ(t-tbar), these limits were then converted into limits on the Branching Ratios for each of studied FCNC top decay channels.

35. Workshop,Tbilisi,18/2005 The expected 95% confidence level limits on the FCNC top decays branching ratio in the absence of signal hypothesis AnalysisLeptonic Mode Hadronic Mode Combined Result Sequential 6.3 x 10 -5 2.6 x 10 -4 5.8 x 10 -5 Probabilistic3.4 x 10 -4 6.5 x 10 -5 ---- ---- t→Zu (c) ( L =10 fb -1, L =100 fb -1 )

36. Workshop,Tbilisi,18/2005 The expected 95% confidence level limits on the FCNC top decays branching ratio in the absence of signal hypothesis Analysis t → γu(c) t → g u ( c ) 3jets 4jets combined Sequential --- - Probabilistic 6.6 x 10 -5 1.8 x 10 -5 1.7 x 10 -3 5.0 x 10 -4 2.5 x 10 -3 8.0 x 10 -4 1.4 x 10 -3 4.3 x 10 -4

37. Workshop,Tbilisi,18/2005 3 separate production processes: W-gluon fusion (250 pb) W*/s-channel (10 pb)Wt (60 pb) Electroweak Single Top Quark Production

38. Workshop,Tbilisi,18/2005 Top Quark Rare Decays (cont’d) t  Hq Tblisi group has been studying various approaches to t  Hq Earlier results for t tbar  Hq Wb  (b-bbar)j (l b) for mH = 115 GeV Sensitive to Br(t  Hq) = 4.5 X 10 -3 (100 fb -1 ) New results for t tbar  Hq Wb  WW*q Wb  (l l j) (l b) ≥ 3 isolated leptons with pT(lep) > 30 GeV pTmiss > 45 GeV ≥ 2 jets with pT(j) > 30 GeV, incl. ≥ 1 jet with b-tag kinematic cuts to take advantage of angular and other correlations Sensitive to Br(t  Hq) = 2.4 X 10 -3 for mH = 160 GeV (100 fb -1 )

39. Workshop,Tbilisi,18/2005 Conclusions The Sensitivity of the ATLAS experiment to the FCNC t → X q ( X = Z, γ, g and q=u,c ) decays of the top quark was performed. Different types of analysis ( sequential and probabilistic ) were used to obtain the FCNC branching ratio sensitivities ( assuming a 5σ signal significance for discovery ) or the 95% CL limits on the FCNC branching ratios ( in absence of signal ). In the Sequential analysis the kinematic cuts were applied in sequence for the signal and backgrounds in order to evaluate the expected branching ratio sensitivities. In probabilistic analysis signal and background-like probabilities were computed using Probability Density Functions constructed from relevant physical variables. The results obtained from both types of analysis coincide and are compatible. The expected branching ratio sensitivities obtained by the different analysis are in good agreement, being in the range from 10 -3 to 10 -5 ( for L =100 fb -1 ) The impact of Systematic errors on the final results have been studied Studies of the top quark FCNC decays provide a sensitive window onto possible new physics beyond the SM