1. Visual Inspection Planning with Sensor Constraint Graph
By Alexis H Rivera

2. Overview Definition the problem Nonlinear optimization problem
Visual Inspection
Visual Inspection Planning
Nonlinear optimization problem
Constraint Sensor Graph
Visual Inspection Plan

3. Problem Visual Inspection
Given an object, determine if it satisfies the design specification
Uses camera as measurement tool

4. Problem (cont.) Visual Inspection Planning
Where should the camera be located to optimally inspect the desired geometric entities?
location: camera pose or sensor setting
geometric entities: vertices, edges
need to define optimality

5. Problem (cont.) Optimality criterion
minimize the inherent measurement errors
such that desired entities are:
resolvable
in focus
within field of view
visible
satisfy dimensional tolerances
minimize number of camera sensors needed

6. Measurement Errors Two inherent errors involved in camera placement
Displacement Errors
Quantization Errors

7. Displacement Errors Ideal pose: Actual pose: Consequence:
(tx, ty, tz, Φ, θ, Ψ)
Actual pose:
(tx, ty, tz, Φ, θ, Ψ) + (dtx, dty, dtz, dΦ, dθ, dΨ)
Consequence:
a vertex ideally mapped to image point (u, v)
now is mapped to (u’, v’) causing measurement error

8. Quantization Error
Actual Length:
L = lrx + u + v
Quantized Length

9. Error Model Goal Denote the total error as the R.V. ε = εd + εq
Minimize Mean Square Error
E[ε2] = σε2 + ηε2
E[ε2] = E[εd2] + E[εq2]

10. Resolution For each entity j, there is a constraint g1j()
Map smallest line entity to l pixels
Example: l = 2

11. Focus Two constraints, g2a(), g2b()
Require closest and furthest entity vertices from the camera position to be within the far and near limits of the depth of field
camera rf rc
Far limit
Near limit

12. Field Of View One constraint: g3()
Bounding cone must be contained within the viewing cone
Bounding cone
Viewing cone

13. Visibility Many equations: g4i() for i=1 to m
Plane equations that bound the visibility of the desired entities x y e1 e2 e3 e4
Example:
To see entities e1, e2, e3, e4, the camera must satisfy equation y < 0

14. Visibility Equations are determined from the Entity Aspect Graph
Each node contains visibility equations and entities that are visible on that aspect

15. Visibility (optional)
Equations are determined from the Entity Aspect Graph
Works by finding visual events
Type C
Type A
Type B x y e1 e2 e3 e4

16. Nonlinear optimization program
For a given set of entities, S
minimize E[ε2] = f (tx, ty, tz, Φ, θ, Ψ, S)
subject to:
g1j <= 0 (resolution), for j=1 to k
g2a <= 0 (focus)
g2b <= 0
g3 <= (field of view)
g4i <= 0 (visibility) for i=1 to m

17. Dimensional Tolerance
From optimal pose, the observed dimensions of entities are as a close as possible to the actual real world dimension of the entity
For each line entity
want probability of deviation <= threshold

18. Definitions Passing entity Failing entity Passing optimization
entity whose dimensional error satisfies tolerances
Failing entity
entity whose dimensional error violates tolerances
Passing optimization
All geometric entities are passing entities
Failing optimization
At least one failing entity

19. Inspection Planning (so far)
Determine optimal camera pose for current entity
Determine optimal camera pose for current entity
More entities?
yes
Passing optimization?
Regroup entities
Save camera pose no
yes
How are entity sets determined and regrouped?

20. Sensor Constraint Graph
SCG node: 4 tuple (E,O,G,I)
E: set of desired geometric entities to be observed
O: objective function (MSE)
G = { V, V’}
V visibility constraints
V’ focus, resolution, and field of view constraints
I: initial camera pose

21. Sensor Constraint Graph
SCG Arcs
solid arcs
adjacent, yet disjoint visibility regions between two nodes
dashed arcs
overlapping visibility regions between two nodes

22. SCG operations
Three operations:
Construction
Expansion
Contraction

23. SCG construction Construct an EAG from the set of geometric entities
each node of EAG has set of visible desired entities and visibility constraints
Define objective function for each node
Define sensor constraints G for each node
Choose arbitrary initial camera pose I
Link all nodes with solid arcs

24. Example: SCG construction
{e1,e2} V2
{e3,e2} V3
{e4} V1
EAG
{e1,e2}
G2={V2,V2’},
O2,I2
{e3,e2}
G3={V3,V3’},
O3,I3
{e4}
G1={V1,V1’},
O1,I1
SCG

25. Passing Optimizations
Stored in SLIST
Node of SLIST defined as S={F,E,O}
F camera pose
O objective MSE (value)
E set of entities
SLIST = S

26. Similar Settings Nodes that have identical entity sets
Are combined into new setting So such that
Eo is set of entities
Oo is MSE function
Fo is the camera pose that results in smallest MSE
Similar settings are replaced by So in the SLIST

27. Failing Optimizations
There exist a set of passing entities and failing entities after an optimization
Resolved using expansion and contraction operations

28. SCG expansion Creates subnodes
Desired entity set is a subset of original node
MSE function is defined in terms of such subset
Sensor constraints are also defined in terms of such subsets
Visibility constraints are the same as its original node ??

29. SCG expansion
For multiple subnodes, the union of the desired entity set must be the same as the original nodes’ entity set
All subnodes are connected by dashed arcs 1 2 3
SCG
Expanded SCG 1 2 3a 3b

30. SCG contraction Creates supernodes
Two or more similar nodes are contracted
Supernode has the same entity set as original nodes
Same MSE function
Same sensor constraints
Visibility constraints are the union of the visibility constraints of original nodes
Initial camera pose that results in Min(MSE)

31. SCG contraction
Neighboring nodes keep same relation with respect to supernode
Example: contracting node 1 and 3b
Contracting SCG
1-3b 2 3a
Original SCG 1 2 3a 3b

32. Subnode strategies Two types of failing optimizations
passing entities and failing entities in the set
only failing entities in the set
Five strategies to handle these cases
Strategy 1: Pass/Fail
Strategy 2: One less
Strategy 3: Singleton
Strategy 4: Similar node with one less
Strategy 5: Similar node with singleton

33. Subnode strategies (optional)
Pass/Fail Strategy
create two subnodes, one with passing entities and one with failing entities
One less Strategy
for failing optimization with k entities
create k subnodes containing k-1 entities each
Singleton Strategy
create k subnodes each with 1 entity

34. Subnode strategies (optional)
Similar node with one-less strategy
Select all nodes neighboring No whose entity set is a subset of No’s entity set
Create a subnode that matches the entity set of each neighboring node
If neighbors don’t share any common entities, then use one less strategy
If the union of the neighbors’ entity set doesn’t include all the entities in No, then create an additional subnode with remaining entities

35. Subnode strategies (optional)
Example case 1
Expanded SCG
Contracted SCG
{e1, e2}
{e1,e2,e3}
{e3}
SCG
{e1, e2}
{e1,e2}
{e3}
{e1, e2}
{e3}

36. Subnode strategies (optional)
Example case 2
SCG expanded
{e5, e4}
{e1,e2,e3}
{e6}
SCG
{e1, e2}
{e1,e2}
{e3}
{e1,e3}

37. Subnode strategies (optional)
Example case 3
Expanded SCG
Contracted SCG
{e1, e2}
{e1,e2,e3,
e4}
{e3}
SCG
{e1, e2}
{e1,e2}
{e3}
{e4}
{e1, e2}
{e3}
{e4}

38. Subnode strategies (optional)
Similar nodes with singleton
same as before except using singleton strategy instead of one-less strategy

39. Subnode strategies Choosing between strategies
S1: separates pass/fail entities to consider possibility that failing entities may be observable simultaneously
S2: reduces number of camera poses
S3: reduces computation time
S4 and S5: reduces number of optimizations (supernodes)

40. Inspection Planning (so far)
Determine optimal camera pose for current entity
Determine optimal camera pose for current entity
More entities?
yes
Passing optimization?
Regroup entities
Save camera pose no
yes
How do we process remaining entities?

41. Prioritizing nodes
Prioritize nodes toward greater cardinality in entity sets to reduce number of optimizations
Unprocessed nodes are included in priority queue known as NLIST

42. Prioritizing nodes NLIST = rest(NLIST) NLIST = rest(NLIST)
Priority = 0
NLIST empty?
Priority = #E(N’)
Determine
plan 1
yes
Perform Optimization
on N’ no
N’ = first(NLIST)
done
Passing
optimization?
Save setting
in SLIST
#E(N’) < priority? no
yes no
yes
Create subnodes
Append NLIST
Sort NLIST
Find similar node with E#
combine into supernode No
replace similar nodes with No
sort NLIST
N’ = first(NLIST) 1

43. Inspection Plan The SLIST contains several camera poses Two options
Minimize number of camera poses
Determine smallest total MMSE
Such that all desired entities are observed
Integer program

44. Inspection Plan Let p be the number of camera poses in SLIST
xi = 1 if ith camera pose Si is part of inspection plan for i=1 to p
aij = 1 if entity Ej is in setting Si
n be the total number of desired entities

45. Inspection Plan
Case 2
Case 1

46. Visual Inspection Planning

47. Issues The generation of the visibility constraints is incomplete
The software used to generate such constraints is obsolete
The current nonlinear optimization algorithm is very sensitive to the initial conditions
The process of generating a plan is not automated and is very tedious
The test cases used to validate the software were very simple

48. Research Questions
Will the previous results be affected if a complete entity aspect graph algorithm is used to determine the visibility constraints?
Is there another nonlinear optimization algorithm that is more appropriate for this application?
Is there a way to determine a good initial condition that will guarantee the optimal solution?
Is the current error model good enough? How will the results change if a sub-pixel quantization model is used?
Is the current objective function appropriate? How will the results change if the robustness approach is used instead of the minimum mean square error approach?
Is there a way to characterize the different sub-node strategies?
What observations can be made if objects that are more realistic are used?

49. Research Goal