FM Web Scraping FMPUG: Dallas Chapter Taylor Made Services: FileMaker Presentation March 6, 2009 Dallas Texas

What is FM Web Scraping? FileMaker Web Scraping is the copying of source code from an http file. It is the same thing as going to a web page and viewing the source code. It is also the same thing as copying an http file, which is usually an html or txt file.

Scrape vs. View Use Web Viewer when all you need is to view the information. Use Web Viewer when all you need is to view the information. Scrape a page when you need to store or manipulate data from that page. Scrape a page when you need to store or manipulate data from that page.

Skill Set Needed Understanding of HTML Pages Understanding of HTML Pages Familiarity with Calculations, and in particular, Text Functions Familiarity with Calculations, and in particular, Text Functions Knowledge of Variables Knowledge of Variables Ability to Create Scripts Ability to Create Scripts

Web Viewer Object Create a Layout Create a Layout Add a Web Viewer Object Add a Web Viewer Object Give the Object a Name Give the Object a Name URL Code “scraped” From Web Page Web Page With Object Name

Create URL Open Web Viewer Setup Open Web Viewer Setup Enter URL or URL calculation into the Web Address Enter URL or URL calculation into the Web Address

Create URL Search for desired data in Web Page Search for desired data in Web Page View the Source Code and locate the desired data View the Source Code and locate the desired data Text Calculation Time Text Calculation Time

Do you want to Store the Source Code? You can have it refresh the data each time it loads the web viewer. You can have it refresh the data each time it loads the web viewer. Fresh Data, but doesn’t load quickly Fresh Data, but doesn’t load quickly You can have a calculation or script step save the source code into a text field. You can have a calculation or script step save the source code into a text field. Loads Quickly, but might not be “fresh” data Loads Quickly, but might not be “fresh” data

Text Calculations from Source Code Result can be scraped with: Result can be scraped with: GetLayoutObjectAttribute(“html source” ; “content”) GetLayoutObjectAttribute(“html source” ; “content”) Result can be a direct calculation from the source code Result can be a direct calculation from the source code Delay issues Delay issues Calculate warning if whole page not loaded Calculate warning if whole page not loaded

Challenges Web pages do not load quickly and script steps need delays built into them to allow the load times. Web pages do not load quickly and script steps need delays built into them to allow the load times. Web Pages change! Web Pages change! Build in Warning Calculation Build in Warning Calculation Is the URL available to all users? Is the URL available to all users?

Blank Slide