1. KAGALI PE code development
Hyung Won Lee, Inje University, 29 August 2017
coll. With Jeongcho Kim(Inje U.), Chunglee Kim(KASI)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

2. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Contents
Motivation
Parameter Estimation with Bayesian Inference
Implementation in LALSuite
Strategy for KAGALI
Development for KAGALI
Preliminary test run
Discussions
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

3. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Motivation
Implement MCMC PE for KAGALI
Based on LALSuite
Minimum full functionality for MCMC
Parallel tempered
Improvement as many as possible from LALSuite
Modularize as much as possible for easy maintenance and readable
Test for iKAGRA data
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

4. Parameter Estimation with Bayesian inference
℘ 𝜃 𝑥 = ℘ 𝜃, 𝑥 ℘ 𝑥 = ℘(𝑥|𝜃)℘ 𝜃 ℘(𝑥|𝜃)℘ 𝜃 𝑑𝜃
℘ 𝜃,𝑥 =℘ 𝑥 𝜃 ℘ 𝜃 =℘ 𝜃 𝑥 ℘ 𝑥
𝜃 : unobservable model parameters
𝑥 : observable data
℘ 𝜃,𝑥 : Joint probability observing data 𝑥 with model parameter 𝜃
℘ 𝑥 𝜃 ∝𝐿𝑖𝑘𝑒𝑙𝑖ℎ𝑜𝑜𝑑 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛
Metropolis-Hasting Algorithm, ~ samples
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

5. Parameter Estimation with Bayesian inference
MCMC Samples
PTMCMCOutput.#.nn.h5
PTMCMCOutput.#.h5
Post Process
cbcBayesPostProc.py
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

6. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Waveform Templates
There are many waveform templates
Concentrate TaylorF2 with
Amplitude correction up to 2.5 PN
Effect of eccentricity up to 3.0 PN
TaylorT2 can deal with CBC inspirals when there is a BH spin that is aligned to the orbital angular momentum.
SpinTaylorT4 is one of the most comprehensive inspiral waveform model used by the LIGO-Virgo collaboration. This can deal with precessing CBC inspirals with two spins.
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

7. Likelihood Calculation
Excess energy 𝐸= 1 2 𝑑 𝑑 − 1 2 𝑑−ℎ 𝑑−ℎ = 𝑑 ℎ − 1 2 ℎ ℎ
ℒ(𝑠|𝜃)∝exp 𝐸
Gaussian noise assumption
℘ 𝑥 𝜃 = ℒ(𝑠|𝜃)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

8. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
MCMC Process
Parallel Tempering
Accept next state with probability
𝑟= ℘(𝑥| 𝜃 2 ) 1/𝑇 ℘ 𝜃 ℘(𝑥| 𝜃 1 ) 1/𝑇 ℘ 𝜃 1
Metropolis-Hasting sampling
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

9. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Parallel Tempering
Use few chains with different temperature
Use likelihood ℒ 𝑠 𝜃 1 𝑇 , 𝑇>1
𝑇 𝑚𝑎𝑥 = 𝑁𝑒𝑡𝑤𝑜𝑟𝑘 𝑆𝑁𝑅 2 𝑛 𝑝𝑎𝑟
Improve convergence and mixing
The higher temperature, the smoother distribution
𝑇>1
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

10. Implementation in lalsuite
LAL MCMC Pipeline
Directory structure of lalsuite
Basic entities of LALSuite
LAL applications
Likelihood, proposal and prior functions
Simulation related functions and waveforms
Post Processing python script
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

11. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
LAL MCMC Pipeline
Start
Initialization
MPI synchronize
PT MCMC
Clearance
End
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

12. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
main
LALInferenceInitRunState()
LALInferenceInjectSignal()
LALInferenceApplyCalibrationErrors()
init_ptmcmc()
LALInferenceInitCBCPrior()
LALInferenceInitLikelihood()
LALInferenceDrawThreads()
runState->algorithm
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

13. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
InitRunState()
Makoto & Takuji, −1≈
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

14. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
LALInferenceInjectInspiralSignal()
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

15. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
LALInferenceApplyCalibrationErrors()
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

16. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
init_ptmcmc()
Set Algorithm function pointer
Set parallel swapping function pointer
Set CBC model
Set number of varying parameters
Set temperature ladder
Set CBCthreads if necessary
Set proposal function and proposal arguments for all threads
Set adaptation parameters
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

17. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
LALInferenceInspiralPrior()
Set various prior functions
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

18. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
PTMCMCAlgorithm()
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

19. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Template
LALInferenceTemplateXLALSimInspiralChooseWaveform
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

20. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Strategy for KAGALI
Base on LALSuite for many functions
Modularize if possible
Use structures
Command parameters(KGLProcessParamsTable)
IFO information(KGLInferenceIFOData)
Waveform parameters(KGLWaveformModelParams)
MCMC status(KGLInferenceRunState)
Simulation parameters
Use simple text file for injections
Implement lalinference
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

21. Development for KAGALI
Use a new branch “mcmc” for development
Library and application directory are separated (version 4)
root/kagali/inference/src
root/kagali/kglcommon/clock/src
root/kagali/kglcommon/detector/src
root/kagali/kglcommon/std/src
root/kagali/waveform/src
root/kagaliapps/cbc/mpisrc
Cloned on KISTI and local Ubuntu server
Build and run on both systems
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

22. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Main features
Use a unique union structure for waveform parameters (KGLWaveformModelParams)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

23. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Structures
KGLInferenceIFOData
KGLDetector
KGLWaveformModelParamsValue
KGLWaveformModelParamsType
KGLProcessParamsTable
KGLProcessParamsPair
KGLWaveformModelParamsItem
KGLInferenceAlgorithm
KGLWaveformModelParamsIndex
KGLInferenceRunState
KGLWaveformModelParams
KGLWaveformModelParamsValue
KGLInferenceAlgorithmOneStep
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

24. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

25. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

26. Development for KAGALI
kagali/inference/src
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

27. Development for KAGALI
kagali/kglcommon/clock/src
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

28. Development for KAGALI
kagali/kglcommon/detector/src
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

29. Development for KAGALI
kagali/kglcommon/std/src
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

30. Development for KAGALI
kagali/waveform/src
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

31. Development for KAGALI
kagaliapps/cbc/mpisrc
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

32. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
KGLProcessParamsTable.h.in
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

33. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
KGLProcessParamsTable.h.in
KGLProcessParamsPair
KGLProcessParamsTable
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

34. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
KGLCommandUtils.c
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

35. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
KGLInferenceMCMC.c
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

36. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
inferenceMCMC.c
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

37. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
inferenceMCMC.c
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

38. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Makefile.am
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

39. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
build KAGALI
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

40. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
build script
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

41. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
build script(cont’d)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

42. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Build results
nohup ./build-branch.sh &
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

43. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample Run(condor)
/data/kagra/home/hwlee/projects/LAL/scratch/kagali/mcmc.sub
condor_submit mcmc.sub
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

44. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample Run(condor)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

45. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample Run(condor)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

46. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample Run(condor)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

47. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample Run(condor)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

48. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample run(mpirun)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

49. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Sample run(con’d)
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

50. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Progress from the last
Defines some structures
Waveform Parameters, Datetime, etc.
Implement basic placeholder functions
Implement few more placeholder for functions
Fixing some nil point errors
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

51. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Discussions
Basic data structures are ready
Basic program architecture fixed
Need to write source codes and hurry up
Hanyang, SNU
Seoul
Toyama
KISTI, NIMS
Daejeon
Pusan U.
Osaka
Gimhae
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama

52. KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama
Thanks
KAGALI PE code development, The 17th KAGRA F2F meeting, Toyama