Data Extracts Working Group (DEWG) Update Tuesday, September 25, 2006
Meeting Schedule Last Meeting –September 21, 2006 ERCOT Austin 168 Next Meeting –October 19, 2006 ERCOT Austin 168
DEWG Update Screen Scraping Issues –Concern that there is no other way to get some data without continual website refreshing –Potential data elements missing in certain extracts that is available on website –ERCOT Action: can you determine what is being “hit” the most? ERCOT identified top 3 most hit sites –MP Action: please provide data elements that you think are only available on the website MPs provided many elements they currently hit, ERCOT shared ways to get some of the data, most items are unavailable via extract or API at the frequency required by the market
DEWG Update Screen Scraping Issues –MPs concerned that ERCOT “policies” conflict with current necessity to screen scrape or use continual website refreshing techniques –New ERCOT Action Item:* (possible vote) ERCOT to advise DEWG whether there is a Best Practices associated with “screen scraping” of ERCOT webpages, i.e. this is allowed under specific circumstances. If so, DEWG requests that ERCOT define when MPs can use screen scraping; otherwise advise DEWG how ERCOT can provide information available on ERCOT webpages that is not available in another format – at the same time or at a greater frequency in some instances. –Nodal concerns may need to be addressed
DEWG Update Data Extract Projects –Targeted timeline review –PR EPS Meter Data Extract aka Resource ID Extract –PR 40042_06 Lodestar Batch Extracts –PR EMMS Extracts –PR ESIID Lookup/TDSP ESIID Extract (PRR312) –PR SCR 727 Phase II (ERCOT's move of extract to ODS) –PR SCR 727 Phase II aka SCR740 Web Services –PR EDW Lodestar Transition –PR EDW Siebel Transition –COPS market testing action item
DEWG Update PR Phase II SCR 727 aka SCR 740 Web Services –Progress update –Market Training (November 9 th or 10 th ) –Discuss market questions received by ERCOT – see FAQ document –Review User Guide
DEWG Update MP Testing Concerns –Upcoming Projects MPs will have the opportunity to request ERCOT's verified stabilization files to be FTP’d or mailed –Lodestar Batch Extracts project (PR_40042_06) migrates the Generation, Load, Settlements & Billing, and Shadow Pricing extracts from the Data Archive to the ODS (Operational Data Store). This project is currently in production stabilization and ERCOT is now accepting requests from market participants to receive their verified stabilization files For new extracts Market Test files will continue to be posted to the ERCOT website –Web Services API testing ERCOT is investigating a potential test path for MPs (limited exchange of data) –.NET Environment Concern Raised MP raised a concern that ERCOT should be utilizing.NET environments in testing as part of their testing Best Practices Significant number of MPs use.NET for API related activities DEWG action?
DEWG Update Extract and Report Matrix –The DEWG Extract and Report Matrix is an excellent way to find information, definitions, locations, etc. of a lot of data utilized by MPs: Go to Click Committees and Groups Under COPS Click DEWG Scroll to Key Documents Click DEWG_Extract_and_Report_Matrix
Questions ???