1. SEE-GRID-2 The SEE-GRID-2 initiative is co-funded by the European Commission under the FP6 Research Infrastructures contract no. 031775 Ultra-fast Semiconductor Carrier Transport Simulation on the Grid. SALUTE: New results for the inhomogeneous case Aneta Karaivanova, Emanouil Atanassov and Todor Gurov Institute for Parallel Processing Bulgarian Academy of Sciences Sofia, Bulgaria E-mails:{anet, emanouil,gurov}@parallel.bas.bg

2. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 2 Outline 1. Introduction 2. Monte Carlo Methods 4. Algorithms in SALUTE 5. SEE-GRID Infrastructure 6. Grid Implementation 7. Numerical results 8. Conclusion and future work

3. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 3 Introduction SALUTE (Stochastic ALgorithms for Ultra-fast Transport in sEmiconductors) is a grid application, which integrates a set of novel Monte Carlo and quasi-Monte Carlo algorithms for solving various computationally intensive problems which describe ultrafast carrier transport in semiconductors. SALUTE studies memory and quantum effects during the relaxation process due to electron-phonon interaction in semiconductors. The quantum kinetic model: a femtosecond relaxation process of optically excited carriers in one-band semiconductors or quantum wires. The electron-phonon interaction is switched on after a laser pulse creates an initial electron distribution. Two cases are considered – with and without applied electric field.

4. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 4 SALUTE - Stochastic ALgorithms for Ultra-fast Transport in sEmiconductors Application area:  SALUTE is a grid application developed for solving various computationally intensive problems which describe ultrafast carrier transport in semiconductors. Expected results and their consequences  SALUTE studies memory and quantum effects during the relaxation process due to electron-phonon interaction in semiconductors; present version explores electron kinetics in GaAs nano-wires. Social and scientific impact  Studying the quantum effects that occur at nanometer and femtosecond scale have important scientific (novel advanced methods, investigation of novel physical phenomena) and social impact (by improving the process of designing nano- devices) Potential or existing user or beneficiary community  Scientists and engineers working in the area of nanotechnology from Bulgaria, Macedonia, Austria, UK …

5. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 5 Monte Carlo Methods J is a quantity to be estimated via a MCM (in our application: different functionals of the solution of the integral equation) θ is a random variable with E[θ]=J θ N is the estimator with N samples σ(θ)N -1/2 is the statistical error Reducing the error:  Variance reduction (antithetic variates, control variates, stratification, importance sampling)  Using more powerful random numbers (quasirandom numbers instead of pseudorandom)

6. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 6 Quantum-kinetic equation (inhomogeneous case) The integral form of the equation: Kernels:

7. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 7 Estimated Quantities Wigner function : Energy (or momentum) distribution: Density distribution: Backward time evolution of the numerical trajectories

8. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 8 Algorithms in SALUTE Homogeneous case:  MC algorithm-1 (Zero electric field, Zero temp.);  MC algorithm -2 (with electric field, Zero temp.);  Quasi-MC algorithm (with el. field, different temp.)  Importance Separation MC - 3 (zero el. field and temp.) - new; Inhomogeneous case (Carrier-phonon collisions are non-local in space):  MC algorithm-4 (Zero electric field, Zero temperature)  Hybrid MC algorithm-5 (With electric field, zero temp.) – new  MC algorithm -6 ( with Field, with temperature) - new. O(exp(ct 2 )/N (1/2) ) O(exp(ct)/N (1/2) )

9. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 9 Parallel Monte Carlo Parallelism is an alternative way to accelerate the convergence of a Monte Carlo computation If n processors execute n independent copies of a MC computation, the accumulated result will have a variance n time smaller than that of a single copy A computational grid has attractively tremendous large amount of computational power  Effectively exploring the power of distributed MC application requires that the underlying random number streams in each subtask are independent in a statistical sense  Running quasi-MC in parallel is more challenging because division into subtasks is a non-trivial problem

10. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 10 SALUTE – Infrastructure usage and characteristics Reasons for running the application on the grid  Quantum problems are very computationally intensive.  The inherent parallel nature of MC applications makes efficient use of Grid resources. Grid implementation scheme AMGA – ARDA Metadata Catalogue WMS – Workload Management System CE – Computing Element WN – Worker Node FTS – File Transfer Service SE – Storage Element (dCache or other) GUI – Graphical User Interface Other Core Services – MyProxy, R-GMA, BDII, VOMS, LFC

11. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 11 SALUTE Demonstration Monitoring of jobs and file transfers Visualization of the Wigner function in 3D on the fly Analysis of results

12. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 12 Application Results Results in the inhomogeneous case  Wigner function for evolution time 175 fs  Electron density The results were obtained after 1000-1500 CPU hours The Wigner function without an applied field The Wigner function with an applied field Electron density Comparing Electron density with/without applied field

13. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 13 Application Results Results in the inhomogeneous case  Wave vector  Energy density The results were obtained after running 1000-1500 CPU hours (500 jobs of 2-3 hours each) Wave vector with/without applied electric field Wave vector Energy density

14. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 14 Grid implementation scheme for SALUTE The SALUTE application requires accumulation of results from millions and even billions of trajectories The Grid implementation scheme of SALUTE distributes the work between Grid sites and ultimately Grid WorkerNodes, and utilizes the various Grid Core services in order to manage the job execution and results accumulation. The data about every run is stored for later reuse and comparison, using Storage Elements and AMGA metadata catalogue. We make use of the new services, available in the gLite middleware, for more efficient and robust implementation of the SALUTE computational model, especially in its management of the generated data. For example, the File Transfer Service is used in order to control the usage of networking resources and to avoid overloading of storage elements.

15. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 15 Average peak usage of SEE-GRID by SALUTE Number of jobs – 1000 Mean duration of job – 10 hours Number of used CPU hours – 10000 Number of file transfers – 3000 Total storage used – 30GB Time to finish – 24 hours Sites used – 20

16. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 16 Usage of SEE-GRID by SALUTE Progress in number of jobs completed after submission of 420 jobs to 21 SEE-GRID sites.

17. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 17 BG apps accounting vs some official EGEE appl VOsNormalize CPU time Fusion208 372 Geant4197 445 esr125 805 magic31 092 BG applications12 applications268 439 hours 11 185 days, 30,6 years 150000 hours (SALUTE)

18. 3 rd User Forum, 11-14 February 2008, Clermont-Ferrand 18 Plans for Future Special points allowing good progress  SALUTE is supposed to work on any properly configured SEE- GRID site.  SALUTE can utilize the CPU power of the less resourceful SEE- GRID sites. In order to be able to solve the new applied problems that we face we will need more CPU power  MPI support needs to be improved Future work on SALUTE  Continue to improve the algorithms performance  Polish the User Interface  Implement workaround for MPI on non-MPI enabled sites, that would not require application recompilation