Download presentation
Presentation is loading. Please wait.
1
Cube Pruning as Heuristic Search Mark Hopkins and Greg Langmead Language Weaver, Inc.
2
8/13/20152 Huang and Chiang (2007) Denero et al. (2009) Denero et al. (2009) Moore and Quirk (2007) Iglesias et al. (2009) Roark and Holings head (2008) Li and Khudan pur (2008) Chiang (2007) Zhang and Gildea (2008) Petrov et al. (2008) Pauls and Klein (2009) Pust and Knight (2009) Pust and Knight (2009)
3
8/13/20153 ! distinct terminologies ?> = good ? + specialized use cases
4
8/13/20154 !? #$!%&!!!
5
5 cube pruning cube pruning =A* …on a specific search space …with specific heuristics (Chiang, 2007)
6
8/13/20156 Accuracy (e.g., BLEU) is very important But time is money – for the customer: throughput – for LW: cpu time on servers for SaaS solution Our customers expect 1000-3000 words per minute in one thread – and linear speedup with multiple threads That’s 0.3-1.0 seconds per sentence Can syntax be viable at product speeds? Motivation: Speed vs Accuracy
7
8/13/20157 ilnevapas 01234 [0,1] item1 [0,1] item2 [0,1] item1 [0,1] item2 [1,2] item1 [1,2] item2 [1,2] item3 [1,2] item1 [1,2] item2 [1,2] item3 [2,3] item1 [3,4] item1 [3,4] item2 [3,4] item3 [3,4] item1 [3,4] item2 [3,4] item3 [0,2] item1 [0,2] item2 [0,2] item1 [0,2] item2 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item1 [2,4] item2 [2,4] item3 [1,3] item1 [1,3] item2 [1,3] item1 [1,3] item2 [0,3] item1 [0,3] item2 [0,3] item3 [0,3] item1 [0,3] item2 [0,3] item3 [1,4] item1 [1,4] item2 [1,4] item1 [1,4] item2 Cube pruning targets CKY decoding
![8/13/20157 ilnevapas [0,1] item1 [0,1] item2 [0,1] item1 [0,1] item2 [1,2] item1 [1,2] item2 [1,2] item3 [1,2] item1 [1,2] item2 [1,2] item3 [2,3] item1 [3,4] item1 [3,4] item2 [3,4] item3 [3,4] item1 [3,4] item2 [3,4] item3 [0,2] item1 [0,2] item2 [0,2] item1 [0,2] item2 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item1 [2,4] item2 [2,4] item3 [1,3] item1 [1,3] item2 [1,3] item1 [1,3] item2 [0,3] item1 [0,3] item2 [0,3] item3 [0,3] item1 [0,3] item2 [0,3] item3 [1,4] item1 [1,4] item2 [1,4] item1 [1,4] item2 Cube pruning targets CKY decoding](http://images.slideplayer.com./19/5908570/slides/slide_7.jpg "8/13/20157 ilnevapas [0,1] item1 [0,1] item2 [0,1] item1 [0,1] item2 [1,2] item1 [1,2] item2 [1,2] item3 [1,2] item1 [1,2] item2 [1,2] item3 [2,3] item1 [3,4] item1 [3,4] item2 [3,4] item3 [3,4] item1 [3,4] item2 [3,4] item3 [0,2] item1 [0,2] item2 [0,2] item1 [0,2] item2 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item1 [2,4] item2 [2,4] item3 [1,3] item1 [1,3] item2 [1,3] item1 [1,3] item2 [0,3] item1 [0,3] item2 [0,3] item3 [0,3] item1 [0,3] item2 [0,3] item3 [1,4] item1 [1,4] item2 [1,4] item1 [1,4] item2 Cube pruning targets CKY decoding")
8
8/13/20158 ilnevapas 01234 [1,4] item1 [1,4] item2 [1,4] item1 [1,4] item2 an item encodes a (set of) translation(s) of a partial sentence
![8/13/20158 ilnevapas [1,4] item1 [1,4] item2 [1,4] item1 [1,4] item2 an item encodes a (set of) translation(s) of a partial sentence](http://images.slideplayer.com./19/5908570/slides/slide_8.jpg "8/13/20158 ilnevapas [1,4] item1 [1,4] item2 [1,4] item1 [1,4] item2 an item encodes a (set of) translation(s) of a partial sentence")
9
8/13/20159 How are items created? 2345 [4,5] item1 [4,5] item2 [4,5] item3 [4,5] item1 [4,5] item2 [4,5] item3 [2,3] item1 [2,3] item2 [2,3] item3 [2,3] item1 [2,3] item2 [2,3] item3 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 ????
10
8/13/201510 Items are created by combining items from complementary spans. 2345 [4,5] item1 [4,5] item2 [4,5] item3 [4,5] item1 [4,5] item2 [4,5] item3 [2,3] item1 [2,3] item2 [2,3] item3 [2,3] item1 [2,3] item2 [2,3] item3 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 [2,4] item3 [4,5] item2 [2,5] item1
11
8/13/201511 [ 2,4, NP, the*car ] an item consists of three parts: spanpostconditioncarry What is an item?
![8/13/ [ 2,4, NP, the*car ] an item consists of three parts: spanpostconditioncarry What is an item](http://images.slideplayer.com./19/5908570/slides/slide_11.jpg "8/13/ [ 2,4, NP, the*car ] an item consists of three parts: spanpostconditioncarry What is an item")
12
8/13/201512 an item consists of three parts: spanpostconditioncarry [ 2,4, NP, the*car ] What is an item?
![8/13/ an item consists of three parts: spanpostconditioncarry [ 2,4, NP, the*car ] What is an item](http://images.slideplayer.com./19/5908570/slides/slide_12.jpg "8/13/ an item consists of three parts: spanpostconditioncarry [ 2,4, NP, the*car ] What is an item")
13
8/13/201513 an item consists of three parts: spanpostconditioncarry What is an item? [ 2,4, NP, the*car ]
14
8/13/201514 CKY Item Creation [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] postcondition carry preconditions postcondition cost( subitem1) cost( subitem2) cost( rule ) + interaction( rule, subitems) cost( new item )
![8/13/ CKY Item Creation [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] postcondition carry preconditions postcondition cost( subitem1) cost( subitem2) cost( rule ) + interaction( rule, subitems) cost( new item )](http://images.slideplayer.com./19/5908570/slides/slide_14.jpg "8/13/ CKY Item Creation [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] postcondition carry preconditions postcondition cost( subitem1) cost( subitem2) cost( rule ) + interaction( rule, subitems) cost( new item )")
15
8/13/201515 The Item Creation Problem 2345 [4,5] item1 [4,5] item2 [4,5] item3 [4,5] item1 [4,5] item2 [4,5] item3 [2,3] item1 [2,3] item2 [2,3] item3 [2,3] item1 [2,3] item2 [2,3] item3 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 ???? even if we just store 1000 items here …and we have only 1000s of grammar rules …and 1000 items here 1000100010001000000000 …then item creation can still take a very long time
![8/13/ The Item Creation Problem 2345 [4,5] item1 [4,5] item2 [4,5] item3 [4,5] item1 [4,5] item2 [4,5] item3 [2,3] item1 [2,3] item2 [2,3] item3 [2,3] item1 [2,3] item2 [2,3] item3 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [3,5] item1 [3,5] item2 [3,5] item3 [3,5] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 [2,4] item1 [2,4] item2 [2,4] item3 [2,4] item4 .](http://images.slideplayer.com./19/5908570/slides/slide_15.jpg "even if we just store 1000 items here …and we have only 1000s of grammar rules …and 1000 items here …then item creation can still take a very long time.")
16
8/13/201516 The Item Creation Problem 2345 1000 ???? is there a better way to enumerate the 1000 items of lowest cost for this span without going through the millions of candidate items and taking the best 1000? this is the problem that cube pruning addresses
17
178/13/2015 [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [?,?,?,?] ? [?,?,?,?][?,?,?,?] So far we have: A demonstration of incremental CKY item creation for span [2,5]
![178/13/2015 [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [ , , , ] .](http://images.slideplayer.com./19/5908570/slides/slide_17.jpg " [ , , , ][ , , , ] So far we have: A demonstration of incremental CKY item creation for span [2,5].")
18
18 [2,4] [4,5][2,3] [3,5] 8/13/2015 [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,?,?] [3,5,?,?] ? [2,5,?,?][2,5,?,?] So far we have:
![18 [2,4] [4,5][2,3] [3,5] 8/13/2015 [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3, , ] [3,5, , ] .](http://images.slideplayer.com./19/5908570/slides/slide_18.jpg " [2,5, , ][2,5, , ] So far we have:.")
19
19 [2,4] [4,5][2,3] [3,5] AB 8/13/2015 [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,?] [3,5,?,?] ? [2,5,?,?][2,5,?,?] So far we have:
![19 [2,4] [4,5][2,3] [3,5] AB 8/13/2015 [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A, ] [3,5, , ] .](http://images.slideplayer.com./19/5908570/slides/slide_19.jpg " [2,5, , ][2,5, , ] So far we have:.")
20
8/13/201520 [2,4] [4,5][2,3] [3,5] AB AB [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,?] [3,5,B,?] ? [2,5,?,?][2,5,?,?] So far we have:
![8/13/ [2,4] [4,5][2,3] [3,5] AB AB [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A, ] [3,5,B, ] .](http://images.slideplayer.com./19/5908570/slides/slide_20.jpg " [2,5, , ][2,5, , ] So far we have:.")
21
8/13/201521 [2,4] [4,5][2,3] [3,5] AB AB yn accept rule(A,B,1)? rule(A,B,k) is the kth lowest cost rule whose preconditions are [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,?] [3,5,B,?] B [2,5,B,?][2,5,B,?] So far we have:
![8/13/ [2,4] [4,5][2,3] [3,5] AB AB yn accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_21.jpg "rule(A,B,k) is the kth lowest cost rule whose preconditions are [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A, ] [3,5,B, ] B [2,5,B, ][2,5,B, ] So far we have:.")
22
8/13/201522 [2,4] [4,5][2,3] [3,5] AB AB yn y accept rule(A,B,1)? accept item(2,3,A,1)? item(2,3,A,k) is the kth lowest cost item of span [2,3] whose postcondition is A n [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,a*b] [3,5,B,?] B [2,5,B,?*b] So far we have:
![8/13/ [2,4] [4,5][2,3] [3,5] AB AB yn y accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_22.jpg "accept item(2,3,A,1). item(2,3,A,k) is the kth lowest cost item of span [2,3] whose postcondition is A n [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,a*b] [3,5,B, ] B [2,5,B, *b] So far we have:.")
23
8/13/201523 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn accept rule(A,B,1)? accept item(2,3,A,1)? accept item(3,5,B,1)? y n accept item(3,5,B,2)? [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] So far we have:
![8/13/ [2,4] [4,5][2,3] [3,5] AB AB yn y n yn accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_23.jpg "accept item(2,3,A,1). accept item(3,5,B,1). y n accept item(3,5,B,2). [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] We want: [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b] So far we have:.")
24
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn accept rule(A,B,1)? accept item(2,3,A,1)? accept item(3,5,B,1)? y n accept item(3,5,B,2)? this is a search space The Item Creation Problem, rephrased: find the n lowest- cost goal nodes of this search space 24
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_24.jpg "accept item(2,3,A,1). accept item(3,5,B,1). y n accept item(3,5,B,2). this is a search space The Item Creation Problem, rephrased: find the n lowest- cost goal nodes of this search space 24.")
25
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n so if we can come up with lower-bounds on the best-cost reachable goal node from here… …and here …then we can just run A* on this search space to find the n goal nodes of lowest cost (without searching the entire space) 25
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n so if we can come up with lower-bounds on the best-cost reachable goal node from here… …and here …then we can just run A* on this search space to find the n goal nodes of lowest cost (without searching the entire space) 25](http://images.slideplayer.com./19/5908570/slides/slide_25.jpg "8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n so if we can come up with lower-bounds on the best-cost reachable goal node from here… …and here …then we can just run A* on this search space to find the n goal nodes of lowest cost (without searching the entire space) 25")
26
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n h = -infinity h = greedy lookahead cost 26
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n h = -infinity h = greedy lookahead cost 26](http://images.slideplayer.com./19/5908570/slides/slide_26.jpg "8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n h = -infinity h = greedy lookahead cost 26")
27
8/13/2015 [2,4] [4,5][2,3] [3,5] AB A B h = -infinity 27 accept rule(A,B,1)? yn accept item(3,5,B,1)? yn accept item(2,3,A,1)? ny h = [2,5,B,b*b] cost = 7.48 7.48
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB A B h = -infinity 27 accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_27.jpg "yn accept item(3,5,B,1). yn accept item(2,3,A,1). ny h = [2,5,B,b*b] cost =")
28
8/13/2015 [2,4] [4,5][2,3] [3,5] AB A B h = -infinity 28 accept rule(A,B,1)? yn accept item(3,5,B,1)? y accept item(2,3,A,1)? ny h = 7.48 accept item(2,3,A,2)? yn yn accept item(3,5,B,1)? [2,5,A,a*b] cost = 12.26 h = 12.26
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB A B h = -infinity 28 accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_28.jpg "yn accept item(3,5,B,1). y accept item(2,3,A,1). ny h = 7.48 accept item(2,3,A,2). yn yn accept item(3,5,B,1). [2,5,A,a*b] cost = h =")
29
8/13/2015 [2,4] [4,5][2,3] [3,5] AB A B h = -infinity 29 accept rule(A,B,1)? yn accept item(3,5,B,1)? y accept item(2,3,A,1)? ny h = 7.48 accept item(2,3,A,2)? yn yn accept item(3,5,B,1)? [2,5,A,a*b] cost = 12.26 h =12.26 not a lower bound [2,5,A,a*b] cost = 5.42 7.48 <
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB A B h = -infinity 29 accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_29.jpg "yn accept item(3,5,B,1). y accept item(2,3,A,1). ny h = 7.48 accept item(2,3,A,2). yn yn accept item(3,5,B,1). [2,5,A,a*b] cost = h =12.26 not a lower bound [2,5,A,a*b] cost = <.")
30
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n h = -infinity h = greedy lookahead cost 30 admissible not admissible therefore A* will not find the n best solutions, it will only find n good solutions
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n h = -infinity h = greedy lookahead cost 30 admissible not admissible therefore A* will not find the n best solutions, it will only find n good solutions](http://images.slideplayer.com./19/5908570/slides/slide_30.jpg "8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn y n h = -infinity h = greedy lookahead cost 30 admissible not admissible therefore A* will not find the n best solutions, it will only find n good solutions")
31
8/13/201531 A* search Cube pruning vs.
32
[2,4] [4,5] 8/13/201532 A* search Cube pruning Cube pruning begins by forming cubes [2,3] [3,5] for each choice of subspans and preconditions [2,3] A [3,5] B [2,3] A [3,5] A [2,4] A [4,5] B [2,4] A [4,5] A [2,3] B [3,5] A [2,3] B [3,5] B [2,4] B [4,5] A [2,4] B [4,5] B
![[2,4] [4,5] 8/13/ A* search Cube pruning Cube pruning begins by forming cubes [2,3] [3,5] for each choice of subspans and preconditions [2,3] A [3,5] B [2,3] A [3,5] A [2,4] A [4,5] B [2,4] A [4,5] A [2,3] B [3,5] A [2,3] B [3,5] B [2,4] B [4,5] A [2,4] B [4,5] B](http://images.slideplayer.com./19/5908570/slides/slide_32.jpg "[2,4] [4,5] 8/13/ A* search Cube pruning Cube pruning begins by forming cubes [2,3] [3,5] for each choice of subspans and preconditions [2,3] A [3,5] B [2,3] A [3,5] A [2,4] A [4,5] B [2,4] A [4,5] A [2,3] B [3,5] A [2,3] B [3,5] B [2,4] B [4,5] A [2,4] B [4,5] B")
33
h = -inf 8/13/201533 A* search Cube pruning Cube pruning begins by forming cubes [2,3] B [3,5] A [2,3] A [3,5] B A* search visits nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB h = -inf -inf therefore, it will begin by visiting all nodes with –inf heuristics [2,3] A [3,5] A [2,3] B [3,5] B [2,4] A [4,5] B [2,4] A [4,5] A [2,4] B [4,5] A [2,4] B [4,5] B
![h = -inf 8/13/ A* search Cube pruning Cube pruning begins by forming cubes [2,3] B [3,5] A [2,3] A [3,5] B A* search visits nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB h = -inf -inf therefore, it will begin by visiting all nodes with –inf heuristics [2,3] A [3,5] A [2,3] B [3,5] B [2,4] A [4,5] B [2,4] A [4,5] A [2,4] B [4,5] A [2,4] B [4,5] B](http://images.slideplayer.com./19/5908570/slides/slide_33.jpg "h = -inf 8/13/ A* search Cube pruning Cube pruning begins by forming cubes [2,3] B [3,5] A [2,3] A [3,5] B A* search visits nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB h = -inf -inf therefore, it will begin by visiting all nodes with –inf heuristics [2,3] A [3,5] A [2,3] B [3,5] B [2,4] A [4,5] B [2,4] A [4,5] A [2,4] B [4,5] A [2,4] B [4,5] B")
34
8/13/201534 A* search Cube pruning Cube pruning begins by forming cubes A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3] A [3,5] B [2,3] B [3,5] A [2,3] B [3,5] B [2,3] A [3,5] A [2,4] A [4,5] B [2,4] B [4,5] A [2,4] A [4,5] A [2,4] B [4,5] B
![8/13/ A* search Cube pruning Cube pruning begins by forming cubes A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3] A [3,5] B [2,3] B [3,5] A [2,3] B [3,5] B [2,3] A [3,5] A [2,4] A [4,5] B [2,4] B [4,5] A [2,4] A [4,5] A [2,4] B [4,5] B](http://images.slideplayer.com./19/5908570/slides/slide_34.jpg "8/13/ A* search Cube pruning Cube pruning begins by forming cubes A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3] A [3,5] B [2,3] B [3,5] A [2,3] B [3,5] B [2,3] A [3,5] A [2,4] A [4,5] B [2,4] B [4,5] A [2,4] A [4,5] A [2,4] B [4,5] B")
35
8/13/201535 A* search Cube pruning What is a cube? A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3] A [3,5] B a cube is a set of three axes
36
8/13/201536 A* search Cube pruning What is a cube? A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) [2,3] A [3,5] B sorted by increasing cost [2,3] A [3,5] B sorted by increasing cost [2,3] A [3,5] B
37
8/13/201537 A* search Cube pruning Thus each choice of object from A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) here item(2,3,A,2) here item(3,5,B,1) and here rule(A,B,4)
![8/13/ A* search Cube pruning Thus each choice of object from A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) here item(2,3,A,2) here item(3,5,B,1) and here rule(A,B,4)](http://images.slideplayer.com./19/5908570/slides/slide_37.jpg "8/13/ A* search Cube pruning Thus each choice of object from A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) here item(2,3,A,2) here item(3,5,B,1) and here rule(A,B,4)")
38
8/13/201538 A* search Cube pruning …creates a new item for [2,5] A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) item(2,3,A,2) item(3,5,B,1) rule(A,B,4) [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b]
![8/13/ A* search Cube pruning …creates a new item for [2,5] A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) item(2,3,A,2) item(3,5,B,1) rule(A,B,4) [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b]](http://images.slideplayer.com./19/5908570/slides/slide_38.jpg "8/13/ A* search Cube pruning …creates a new item for [2,5] A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) item(2,3,A,2) item(3,5,B,1) rule(A,B,4) [2,3,A,a*b] [3,5,B,b*a] B [2,5,B,b*b]")
39
8/13/201539 A* search Cube pruning If we take the best representative from each axis (i.e. the “1-1-1”)… A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) …then we expect the resulting item to have a low cost, since: cost( new item ) = cost( subitem1 ) + cost( subitem2 ) + cost( rule ) + interaction( rule, subitems )
40
8/13/201540 A* search Cube pruning Though we are not guaranteed this, because: A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) cost( new item ) = cost( subitem1 ) + cost( subitem2 ) + cost( rule ) + interaction( rule, subitems ) this cost is not monotonic
![8/13/ A* search Cube pruning Though we are not guaranteed this, because: A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) cost( new item ) = cost( subitem1 ) + cost( subitem2 ) + cost( rule ) + interaction( rule, subitems ) this cost is not monotonic](http://images.slideplayer.com./19/5908570/slides/slide_40.jpg "8/13/ A* search Cube pruning Though we are not guaranteed this, because: A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB [2,3]A [3,5]B item(2,3,A,1) item(2,3,A,2) item(2,3,A,3) item(3,5,B,3) item(3,5,B,2) item(3,5,B,1) rule(A,B,1) rule(A,B,2)rule(A,B,3)rule(A,B,4) cost( new item ) = cost( subitem1 ) + cost( subitem2 ) + cost( rule ) + interaction( rule, subitems ) this cost is not monotonic")
41
8/13/201541 A* search Cube pruning A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the 1-1-1 item of every cube 2.44.67.93.2 5.59.26.24.4 and scoring them
![8/13/ A* search Cube pruning A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the item of every cube and scoring them](http://images.slideplayer.com./19/5908570/slides/slide_41.jpg "8/13/ A* search Cube pruning A* search visits nodes in order of increasing heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the item of every cube and scoring them")
42
8/13/201542 A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the 1-1-1 item of every cube and scoring them accept rule(A,B,1) accept item(2,3,A,1) accept item(3,5,B,1) 111 2.4 4.67.93.2 5.59.26.24.4
![8/13/ A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the item of every cube and scoring them accept rule(A,B,1) accept item(2,3,A,1) accept item(3,5,B,1)](http://images.slideplayer.com./19/5908570/slides/slide_42.jpg "8/13/ A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the item of every cube and scoring them accept rule(A,B,1) accept item(2,3,A,1) accept item(3,5,B,1)")
43
8/13/201543 A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the 1-1-1 item of every cube 2.44.67.93.2 5.59.26.24.4 and scoring them 2.44.64.46.23.27.95.59.2
![8/13/ A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the item of every cube and scoring them](http://images.slideplayer.com./19/5908570/slides/slide_43.jpg "8/13/ A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 Cube pruning proceeds by creating the item of every cube and scoring them")
44
8/13/201544 A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 At this point, cube pruning takes the best item it has created. 2.44.67.93.2 5.59.26.24.4 It keeps this item. 2.44.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6
![8/13/ A* search Cube pruning Meanwhile, A* search has scored its frontier nodes [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 At this point, cube pruning takes the best item it has created.](http://images.slideplayer.com./19/5908570/slides/slide_44.jpg "It keeps this item KEPT And generates its one-off items")
45
8/13/201545 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 2.4
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_45.jpg "KEPT And generates its one-off items")
46
8/13/201546 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn 211111 2.44.8
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_46.jpg "KEPT And generates its one-off items yn")
47
8/13/201547 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn 2.44.8
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_47.jpg "KEPT And generates its one-off items yn")
48
8/13/201548 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn 4.8 yn 121111 2.47.1
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_48.jpg "KEPT And generates its one-off items yn 4.8 yn")
49
8/13/201549 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn 4.8 yn 2.47.1
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_49.jpg "KEPT And generates its one-off items yn 4.8 yn")
50
8/13/201550 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn 4.8 yn 7.1 yn 112111 2.43.6
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_50.jpg "KEPT And generates its one-off items yn 4.8 yn 7.1 yn")
51
8/13/201551 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn 4.8 yn 7.1 yn 2.43.6 this is a goal node
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_51.jpg "KEPT And generates its one-off items yn 4.8 yn 7.1 yn this is a goal node.")
52
8/13/201552 A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB 111 4.67.93.2 5.59.26.24.4 It keeps this item. 4.64.46.23.27.95.59.2 KEPT And generates its “one-off” items. 211 121112 4.8 7.13.6 yn yn yn 4.8 7.1 3.6 this is a goal node 111 So A* keeps this item. KEPT
![8/13/ A* search Cube pruning A* search continues to visit nodes in increasing order of heuristic value [2,4] [4,5][2,3] [3,5] ABAB AB A B A B AB It keeps this item.](http://images.slideplayer.com./19/5908570/slides/slide_52.jpg "KEPT And generates its one-off items yn yn yn this is a goal node 111 So A* keeps this item. KEPT.")
53
8/13/201553 A* search Cube pruning = = almost (+ node tying) = =
54
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn yn h = -infinity h = greedy lookahead cost 54 accept rule(A,B,1)? accept item(2,3,A,1)? accept item(3,5,B,2)? accept item(3,5,B,1)? ? Cube pruning was specifically designed for hierarchical phrase-based MT which uses only a small number of distinct postconditions But say our use case was string-to-tree MT, in the style of (Galley et al 2006)
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn yn h = -infinity h = greedy lookahead cost 54 accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_54.jpg "accept item(2,3,A,1). accept item(3,5,B,2). accept item(3,5,B,1). Cube pruning was specifically designed for hierarchical phrase-based MT which uses only a small number of distinct postconditions But say our use case was string-to-tree MT, in the style of (Galley et al 2006).")
55
8/13/201555 Average number of search nodes visited, per sentence Nodes by TypeCube Pruning subspan12936 precondition851458 rule33734 item119703 goal74618 TOTAL1092449 BLEU38.33 Arabic-English NIST 2008 the early nodes with infinite heuristics dominate the search time
56
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn yn h = -infinity h = greedy lookahead cost 56 accept rule(A,B,1)? accept item(2,3,A,1)? accept item(3,5,B,2)? accept item(3,5,B,1)? ? h = something better !
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn yn h = -infinity h = greedy lookahead cost 56 accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_56.jpg "accept item(2,3,A,1). accept item(3,5,B,2). accept item(3,5,B,1). h = something better !.")
57
Nodes by TypeCube PruningAugmented CP subspan1293612792 precondition851458379954 rule3373433331 item119703118889 goal7461874159 TOTAL1092449619125 BLEU38.3338.22 8/13/201557 Number of search nodes visited Nodes by TypeCube Pruning subspan12936 precondition851458 rule33734 item119703 goal74618 TOTAL1092449 BLEU38.33 Arabic-English NIST 2008
58
8/13/201558 Tradeoff curves Arabic-English NIST 2008
59
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn yn h = -infinity h = greedy lookahead cost 59 accept rule(A,B,1)? accept item(2,3,A,1)? accept item(3,5,B,2)? accept item(3,5,B,1)? h = admissible Cube pruning becomes exact. We found that our exact version of cube pruning had a similar time/quality curve to the original inexact version of cube pruning. However it was not as effective as our “augmented” version of cube pruning. This is all interesting, but somewhat beside the point.
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn yn h = -infinity h = greedy lookahead cost 59 accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_59.jpg "accept item(2,3,A,1). accept item(3,5,B,2). accept item(3,5,B,1). h = admissible Cube pruning becomes exact. We found that our exact version of cube pruning had a similar time/quality curve to the original inexact version of cube pruning. However it was not as effective as our augmented version of cube pruning. This is all interesting, but somewhat beside the point..")
60
8/13/201560 What does cube pruning teach us? esson
61
8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn accept rule(A,B,1)? accept item(2,3,A,1)? accept item(3,5,B,1)? y n accept item(3,5,B,2)? this is a search space 61 It tells us that it is useful to frame the CYK Item Generation Problem as a heuristic search problem. Once this realization is made, we suddenly have many more avenues available to us, when implementing a CKY decoder for a particular use case
![8/13/2015 [2,4] [4,5][2,3] [3,5] AB AB yn y n yn accept rule(A,B,1).](http://images.slideplayer.com./19/5908570/slides/slide_61.jpg "accept item(2,3,A,1). accept item(3,5,B,1). y n accept item(3,5,B,2). this is a search space 61 It tells us that it is useful to frame the CYK Item Generation Problem as a heuristic search problem. Once this realization is made, we suddenly have many more avenues available to us, when implementing a CKY decoder for a particular use case.")
62
8/13/201562 h = -infinity h = greedy lookahead cost h = -infinityh = something better We can change the heuristics.
63
8/13/201563 We can change the search algorithm. We can change the heuristics.
64
8/13/201564 We can change the search algorithm. For instance, instead of A*…
65
8/13/201565 We can change the search algorithm. For instance, instead of A*… …we could try a depth-first strategy like depth-first branch- and-bound, and take advantage of its anytime properties.
66
8/13/201566 We can change the search algorithm. We can change the search space.
67
8/13/201567 What does cube pruning teach us? esson We end up with a speedup technique which is: simple general well-studied easily adaptable to new use cases
68
8/13/201568 Thank you. Questions?
Similar presentations
