Automating the Extraction of Data Behind Web Forms Automating the Extraction of Data Behind Web Forms by Sai Ho Yau Brigham Young University
NextPreviousIntroduction There are enormous amounts of information available from the Web, but it is difficult to extract the data automatically due to several reasons: Web information is stored in databases Form interfaces Relevant information can be obtained only after a Web form is filled out and submitted
NextPrevious Problems Dealing with Forms No general Web form design Required text fields One form may lead to another Resulting information embedded within forms Returned error messages versus valid data Elimination of possible duplicate data
NextPrevious The Framework
NextPreviousTools Language and Internet browser used: JavaScript, Java, PHP, MySQL; Microsoft Internet Explorer Platform: Solaris Intel (Unix), with Sun Java.
NextPrevious Method: Construct the Query String
NextPrevious Method: Construct the Query String
NextPrevious The Goal Fills in HTML forms Retrieves data Eliminates duplicates Automatically extract data behind Web forms The system:
NextPrevious Returned Web Page
NextPrevious Suggested Solution Two phases to deal with many possible responses to a query*: Sampling phase Exhaustive phase * Assuming no HTTP error
NextPrevious Sampling Phase Submit the default form. Randomly select N form-field settings and submit the form N times. If no new information, STOP and send the result downstream (N is set so that the probability of subsequent submissions yielding new data is less than 5%). Otherwise, ENTER the Exhaustive Phase.
NextPrevious Exhaustive Phase Estimate the total time and quantity of data. If below threshold, exhaustively obtain the rest of the information. Otherwise, return the results of the sampling and report to the user the estimate of time and quantity of data.
NextPrevious Data Retrieving Strategy Locate possible duplicate information from subsequent retrieved Web pages during Sampling and Exhaustive Phases. Discard duplicates and merge new information. Send fully merged data downstream.
NextPreviousConclusions Eliminate duplicate data and merge resulting information. We can automatically: Fill in Web forms. Extract information behind forms. Screen out errors.