Visual Relations, Part 2 Advanced Visual Analysis

Agenda Visual Relations & Index How Text Analysis works –Do we have a match? –OCR assumptions –My text is not found, what should I do?

Part 1 Visual Relations

Index –The system by default selects the first occurrence of a string on the screen. In case the needle has multiple occurrences, enter the index of the required occurrence.

Example of When to Use Visual Relations The Need –We would like to click on the “Free” button of the PayPal application The Problem(s) 1.The “Free” button is not unique on the screen 2.We can’t assume that the PayPal application download will always remain as the fourth one in the listing. The Solution: –Using Visual Relations! First Needle Second Needle

How to Use Visual Relations - Instructions Using the example on the previous page Step 1: Find the First Needle The Find command searches for an image or text and stores it in memory for the second command. In our example, use the Find function to search the screen for the PayPal icon. The system will store the area coordinates of the first object. Step 2: Use Visual Relations in Select, Button.Click, Edit commands 1.Declare a relationship to the first needle by enabling the visual relation direction and inline command parameters. 2.Go to the 'VR' window to simulate the object search by defining the first object to be found (this must be simulated, since the real object will be found during the live run). 3.A unique, dynamic haystack will be displayed. 4.Return to the snapshot window. 5.Click Try to complete a test search.

Understanding the Relationship Between Needles What does this mean? –The First Needle, which is the application image is... Left and Horizontal –to the Second Needle, which is the 'Buy Now' button VR Window Dynamic Haystack Relations Snapshot Window

ByMobileVisualRelation This class is used for visual relations The visualRelation() method gets two By arguments: 1.The first one is for finding the reference element 2.The second one is for finding the target element WebElement element = visualDriver.findElement(ByMobile.visualRelation(By.linkText("call"), By.linkText("message"))); element.click(); /* * This example shows how to use visual relations; left is the default direction and horizontal is the default inline. */ IMobileWebDriver visualDriver = device.getVisualDriver(); MobileVisualRelations visualRelations = visualDriver.manageMobile().visualOptions().genericOptions().visualRelations(); visualRelations.setDirection(MobileVisualRelationDirection.LEFT); visualRelations.setInline(MobileVisualRelationInline.HORIZONTAL); WebElement element = visualDriver.findElement(ByMobile.visualRelation(By.linkText("Chrome"), By.linkText("Click"))); element.click();

Part 2 How Text Analysis Works - OCR

Text Analysis - How It works System finds text on screen and compares it to the needle (required text) OCR is not an exact science so comparison allows for imperfect matches –The number of characters that are “forgiven” is relative to the length of the string NeedleThreshold %Characters Example: Using the table to the left, a text containing 10 characters can have up to 3 characters wrong and still be found. Example: Using the table to the left, a text containing 10 characters can have up to 3 characters wrong and still be found.

Best Practices Always type in the full string you are searching for When a 100% match is required, use the Threshold parameter to override the system's calculation –For example, where there are similar strings on the screen (Row1, Row2)

More Text Analysis info How we analyze text, by default –White Spaces are ignored –Punctuation is ignored –Capitalization is ignored –It is possible to override when needed Whole Words – accept a word within a word –For example, define whether “person” in “personal” is a good match or not –Accepted by default

Finding the text on the screen - OCR Turn on OCR Profiles Try all the profiles with the word accuracy –Per profile, try changing the DPI –Default is 96, try using 300, then 120

Resources Javadoc – Visual Relations – Video & How to Guide – https://community.perfectomobile.com/series/ Code snippets – mobilecloud-webdriver-code-snippets mobilecloud-webdriver-code-snippets GitHub repository –

Thank You