Texan Streamline Processing (TSP) Eliminating the intermediate step in converting Texan raw data to standard SEG-Y. Steven Harder Presented at the Active.

Slides:



Advertisements
Similar presentations
Sorting Really Big Files Sorting Part 3. Using K Temporary Files Given  N records in file F  M records will fit into internal memory  Use K temp files,
Advertisements

Out-of core Streamline Generation Using Flow-guided File Layout Chun-Ming Chen 788 Project 1.
Avishai Wool lecture Introduction to Systems Programming Lecture 8 Input-Output.
Parallel Geoscience Corporation
Fundamentals Introduction Seismic waves: Propagation Velocity and Amplitudes Seismogram Measurement systems Sources, receivers, Acquisition strategies.
1. visit to IRIS PAGE from
FALL 2004CENG 351 Data Management and File Structures1 External Sorting Reference: Chapter 8.
FALL 2006CENG 351 Data Management and File Structures1 External Sorting.
Manajemen Basis Data Pertemuan 2 Matakuliah: M0264/Manajemen Basis Data Tahun: 2008.
CPSC 231 Sorting Large Files (D.H.)1 LEARNING OBJECTIVES Sorting of large files –merge sort –performance of merge sort –multi-step merge sort.
Polar Coordinates (MAT 170) Sections 6.3
Using Secondary Storage Effectively In most studies of algorithms, one assumes the "RAM model“: –The data is in main memory, –Access to any item of data.
Chapter 8: I/O Streams and Data Files. In this chapter, you will learn about: – I/O file stream objects and functions – Reading and writing character-based.
Computer Hardware In this lecture, we will study:
13.1 Understanding Files The File class Objects can read and write to the file system Use the File class to hold information about files and directories.
Computer Architecture Part III-C: Memory Access and Management.
Storing Data. Memory vs. Storage Storage devices are like file drawers, in that they hold programs and data. Programs and data are stored in units called.
Emlyn Corrin, DPNC, University of Geneva EUDAQ Status of the EUDET JRA1 DAQ software Emlyn Corrin, University of Geneva 1.
Computers Central Processor Unit. Basic Computer System MAIN MEMORY ALUCNTL..... BUS CONTROLLER Processor I/O moduleInterconnections BUS Memory.
Input/OUTPUT [I/O Module structure].
HARDWARE: CPU & STORAGE How to Buy a Multimedia Computer System.
2.0 Computer System.
Contents HADOOP INTRODUCTION AND CONCEPTUAL OVERVIEW TERMINOLOGY QUICK TOUR OF CLOUDERA MANAGER.
Storage Manager Overview L3 Review of SM Software, 28 Oct Storage Manager Functions Event data Filter Farm StorageManager DQM data Event data DQM.
3DMARINE GEOMETRY OVERVIEW of PROCESSING METHODOLOGY.
THE LWA SOFTWARE LIBRARY Jayce Dowell – LWA Users’ Meeting – July 27, 2012.
Database Management Systems,Shri Prasad Sawant. 1 Storing Data: Disks and Files Unit 1 Mr.Prasad Sawant.
CPSC 404, Laks V.S. Lakshmanan1 External Sorting Chapter 13: Ramakrishnan & Gherke and Chapter 2.3: Garcia-Molina et al.
Programming for GCSE Topic 5.1: Memory and Storage T eaching L ondon C omputing William Marsh School of Electronic Engineering and Computer Science Queen.
Introduction to z/OS Basics © 2006 IBM Corporation Chapter 7: Batch processing and the Job Entry Subsystem (JES) Batch processing and JES.
Sorting by the Numbers Sorting Part Four. Question Suppose you are given the task of writing an application to sort a big data file. What do you need.
CS4432: Database Systems II Query Processing- Part 2.
1 5. Abstract Data Structures & Algorithms 5.6 Algorithm Evaluation.
Jay Lofstead Input/Output APIs and Data Organization for High Performance Scientific Computing November.
LCD simulation software efforts at NIU Dhiman Chakraborty Northern Illinois University (NIU)/ Northern Illinois Center for Accelerator.
IT Essentials Chapter 1 Part C. 1.5 Math for a Digital Age.
MTEM data formats What data do we store. Outline Images of the MTEM dictionary – a dialect of the SEGY. Comments on data files.
11.1 Silberschatz, Galvin and Gagne ©2005 Operating System Principles 11.5 Free-Space Management Bit vector (n blocks) … 012n-1 bit[i] =  1  block[i]
Introduction to Database Systems1 External Sorting Query Processing: Topic 0.
Lecture Overview Shift Register Buffering Direct Memory Access.
CS4432: Database Systems II
Using marine air gun shots recorded by onshore broadband instruments: Example from STEEP project, Southern Alaska Christeson et al., JGR, 2013.
Computer Programming II Lecture 9. Files Processing - We have been using the iostream standard library, which provides cin and cout methods for reading.
2.3 Output Formatting. Outputting Format Specify the number of spaces, “c”, used to print an integer value with specifier %cd, e.g., %3d, %4d. E.g. printf.
1 5. Abstract Data Structures & Algorithms 5.6 Algorithm Evaluation.
Chapter 3 Data Representation
CHAPTER 10: Computer Peripherals
ALICE experience with ROOT I/O
CS522 Advanced database Systems
Lecture 16: Data Storage Wednesday, November 6, 2006.
Database Management Systems (CS 564)
Cross-line shot interpolation
HDF5 Metadata and Page Buffering
Lecture Topics: 11/1 Processes Process Management
TerraForm3D Plasma Works 3D Engine & USGS Terrain Modeler
CSCI 315 Operating Systems Design
Binary Files.
Database Management Systems (CS 564)
Lecture#12: External Sorting (R&G, Ch13)
Structured Program Design
I/O Systems I/O Hardware Application I/O Interface
Five Key Computer Components
Standard Grade Revision
File Storage and Indexing
RDBMS Chapter 4.
Chapter 2 Processing Data Peter Norton’s  Introduction to Computers Data Useful output.
Little Man Computer There’s a little man in the mailroom that follows each instruction to the letter but he can only follow one instruction at a time.
File System Implementation
Networks & I/O Devices.
Presentation transcript:

Texan Streamline Processing (TSP) Eliminating the intermediate step in converting Texan raw data to standard SEG-Y. Steven Harder Presented at the Active Source Data Management Workshop Jan , 2014.

Data Flow in TSP Read metadata dasfile, shot, geometry files Read 1 Texan raw file Convert & Write 1 trace w/ timing corrections, geometry and offsets to each SEG-Y file

The Big Surprise TSP runs approximately 30 times faster (clock time) than the two-step 125segy/texan2segy combination. The primary reason for this speed is that both data input and output are read and written in large blocks. Input is read one 528-byte page at a time. Output is written one trace with header at a time.

Tape vs. Disk I/O Sequential vs. Random Access

Advantages of TSP Speed: ~30 times faster than two-step methods using sequential access. Allows for the additional of traces to existing SEG-Y files as raw data become available. Uses a relational data with natural organization of data. Uses either rectangular and spherical coordinate systems.

Future Directions Output both SEG-Y and PH5 formats simultaneously? Raw Data Dasfile Geometry Shot or Event File TSP Standard SEG-Y files Ready to plot PH5 file Ready to archive ?? TSP is available at