1. US≡Universe of Scalar pTreeSets A ScalarpTreeSet is the complete set of pTrees for a column of real numbers (Complete:  a pTree  bit pos, - to 
ToC entry for a SpTSs in a RBaDB
Real Bitarray DataBase
predicate? level=? depth=? Min=?
Rank¼ Median=Rank½? Rank¾>
Max=? Sum=?
bit pos ptr|purity 1count
(n+1,) pure0 0
n pointern
n-1 pointern
...
n' pure1 depth
n" pure0 0
-m pointer-m
(-,-m-1) pure0
Notes:
0. pointerk =pointer to a bit vector.
1.1st and last rows can be implied?
2. Other cols? (e.g., Identical Twins)
3. Need separate ToC for PTreeSets
(as sequences of same-depth SpTSs for tables of real numbers).
4. It's OK to black box SpTSs as real columns since then complex columns can be defined at a higher ToC level via pointers to their real and complex parts.
5. To mult by 2k shift bit pos defs only.
So a cleaner ToC might be:
How do we define (black box) ScalarpTreeSets (SpTSs) in a ToC or VDB Catalog?
n-ary operations: US...USUS
|SpTSresult| ≡ depth = min{|SpTS1|,..., |SpTSm|}
add (row-wise) SpTS SpTSm  SpTSresult SpTSresult,k=SpTS1,k+..+SpTSm,k
-, /, * are similarly defined row_wise operations.
=, >, <, ,  are binary ops producing mask pTree (i.e., bitwidth(SpTSresult)=1
SPc=Scalar Product (multiplying every row of an SpTS by a constant, c). One can use * above or possibly, use c's bit pattern to avoid constructing a constant SpTS)
DPv:=Dot Product with a fixed real vector, vV Again, use v's bit pattern?
SDv=Square Distance from fixed real vector, vV Again use v's bit pattern only?
ERa= EinRings = pTree mask of rows < a apply < above? Better, use a's bit pattern only?
AGa = YC's Aggregates, count, sum, avg, max, min, median, rank_k, top_k, IceBergQueries (Here, the result is a number, but a number is a depth=1, width=1 SpTS.)
ToC for SpTSs in a RBaDB
predicate? .. Sum?
Highest_NonPure0_Bit_Position=n
PointerArray=( ptr(n),...,ptr(-m) )
CountArray=( cnt(n),...,cnt(-m) )
I.e., use offsets instead of keywords to implement the pTree pointer table.
Some red info is redundant. How much pre-computed (redundant?) info should be placed in the ToC?
Rules of thumb: Pre-compute everything that might be useful and certainly pre-compute all info that might require Horizontal Processing of Vertical pTrees. I believe Min, Med, Max, Sum can be derived from ToC info (the counts) without accessing actual pTrees and thus, should be store in the ToC only if doing so save significant processing time.

2. Oblique FAUST with a comprehensive initial linear step (done in parallel?)
I have not used column numbers. e.g., n=minimum means n is a number and it is
the minimum of a column of numbers (indicated here by the position in which it
appears, not by subscript (offset, not keyword identification of the column)).
Each of n,x,a,m are width=4 vectors or 4-tuples of numbers, so they are:
(n1, n2, n3, n4)
For table,Y=(Y1.Y2.Y3.Y4), let n=minYk,
x=maxYk a=avgYk m=medianYk k=1|2|3|4
yY, Lpq(y)=(y-p)o(q-p/|p-q|), p,q any of
p and q form diagonals:
nnnn - xxxx
nnnx - xxxn
nnxn - xxnx
nnxx - xxnn
nxnn - xnxx
nxnx - xnxn
nxxn - xnnx
nxxx - xnnn
xnnn - nxxx
xnnx - nxxn
xnxn - nxnx
xnxx - nxnn
xxnn - nnxx
xxnx - nnxn
xxxn - nnnx
xxxx - nnnn
(x1, x2, x3, x4)
(a1, a2, a3, a4)
(m1, m2, m3, m4)
As on slide 1, these are precomputed and stored as ToC info 1 2 3 4 5 6 7 8 9 a b c d e f
yY = PTreeSet for a table of reals, for any of the 44 (p,q), then in a first dot product with (q-p) step do all 44 dot products and gap analyses! Lot of work? Yes, but there is computational parallelism (Note there is no need to insist on unit vectors - we can dot with q-p and then realize that the gaps will be |q-p| wider than they would have been, had we used dpq instead (affects choice of threshold only)). So if p=(n1, x2, n3, x4) then q=(x1, n2, x3, n4), and then
(y-p)o(q-p)=yoq-yop-poq+pop= (y1y2y3y4)o(n1,x2,n3,x4)-
(y1y2y3y4)o(x1,n2,x3,n4)-poq+pop
n1y1+
x2y2+
n3y3+
x4y4+
x1y1+
n2y2+
x3y3+
n4y4
precomputed fixed vectorss
8 thru f are
the same
diagonals as
7 thru 0,
so we only
need 2n-1 =8
0 thru 7
So if, given Y pre-compute all these scalar-times-SpS binary multiplications of Yk with the 16 pre-computed numbers, nk, xk, ak and mk, then any of these dot product SpSs is a 8-sum of those we precomputed and is thus not much work (4*45= sums).
It may be possible to engineer 8-summing process to cut time even further? Or to engineer an efficient way to do 8 scalar multiplications and the 8-summings in one efficient operation. Of course, it won't always be 8. For the Netflix Movie PTreeSet, there are 17,000 columns, not 4; and for the User PTreeSet there are 500,000.
or p and q midlines
combo(n,n-1) of them
=n!/(n-(n-1))!(n-1)!=n=4
aaan - aaax
aana - aaxa
anaa - axaa
naaa - xaaa
If finding outliers (anomalies), since most anomalies occur at the outside boundaries of a set, this simple method might find all of them without further processing.
The other aspect of OF that needs attention is making this local in the sense that clusters are revealed by local application of linear gap analysis even though for the entire space, no gaps appear. This is the reason to introduce barrel gaps if the initial linear gap analysis fails.
aana - xxxx
xxxn
xnxx
xnxn
nxxx
nxxn
nnxx
nnxn
or p and q from side-mid-pt to a opposite corner (n2n-1 = 32)
aaan -
xxxx
xxnx
xnnx
nxnx
nnnx
anaa - xxxx
xxnn
nxnn
naaa - xxxx
xnnn
By substituting m for a (median for avg) we could get 32 more, but they seem likely to be essentially the same lines as given by a?
Possibility: Always use m instead of a as a center?
32 mdpt-corner are more and more distinct from the diagonals as the dimension increases.
There are (n+1)(2n-1+1) (p,q) pairs to consider.

3. Oblique FAUST a comprehensive initial linear step using a central point, pox mx tx xx xn
For Y=(Y1.Y2.Y3.Y4), n=minYk, x=maxYk
m=medYk o=rank¼Yk t=rank¾Yk k=1|2|3|4
yY, Lpq(y)=yo(q-p)/|q-p|, p=central pt=
the1st pt in the 1st ring around p in SVoM(Y)
q is any of (2n-1 of them)
xxxx
xxxn
xxnx
xxnn
xnxx
xnxn
xnnx
xnnn xt p xm xo
What we are attempting to do here is get a full coverage of ~evenly spaced projection lines (pq-lines or d-lines).
We are doing so by attempting to evenly space q on [half of] the sides.
That seems easier than evenly spacing points on [half of] the sphere (using angles)
If dim=N is high, there may be unevenness to this method?
However, using rank(k/N), k=1..N-1 instead of length increments by =(max-min)/N should ameliorate that?
We can calculate the ranks using our logn pTree rank procedures.
Furthermore, once we move to barrel gapping to limit the radial reach (and therefore materialize gaps for large dataset that would not appear with just linear analysis), using an actual point in the set as p has definite advantages (always centers the barrel on actual points in the space so we get a r=0 radius every time).
or q: m=median pt = Rank½
(n of them)
mmmx
mmxm
mxmm
xmmm
Oblique FAUST 2nd Barrel step (paralleling?)
1. L(q-p)/|q-p|(y)=(y-p)o(q-p)/|q-p| find outliers and linearly gapped clusters. (1/|q-p|=a)
2. Then look for barrel gaps around all dense regions (between thinnings) using
Sp(y)=(y-p)o(y-p)= yoy -2yop +pop
and B(q-p)/|q-p|(y)=Sp(y)-L(q-p)/|q-p|(y)2 = yoy -2yop + pop - (yop-yoq-poq+pop)2/a2
ttxt
ttxo
toxt
toxo
otxt
otxo
ooxt
ooxo
or q: o=Rank¼, t=Rank¾ pt (n2n-1)
tttx
ttox
totx
toox
ottx
otox
ootx
ooox
txtt
txto
txot
txoo
oxtt
oxto
oxot
oxoo
xttt
xtto
xtot
xtoo
xott
xoto
xoot
xooo
Every dot product in the above is a linear combination of SpSs of the form, b*Yk for some b{n.x,a,0,m,t}.
Thus, if we precompute those SpSs we can put together L and B projections efficiently.

4. .b2=p+a2d-pod .q1 .p1 .b1=+a1d-pod Oblique FAUST Pipe 1 d p
1. Linear project onto d-line from within a pipe.
If no good gaps, start over with new p, d, else if good linear gapped region(s) appear,
2. Look for good radial gaps, use them as in OFP0, If none, for each linear gapped region [xod=a1,xod=a2] (corresp. pts on pd-line are bk=p+akd=pod k=1,2), do:
in a narrowed region around p1=avg(b1,b2),
incr radius until 1st gap or thinning appears
(at r1
Let q1=mean(post-thinning barrel stave ring
(radius from r1 to r1 + delta)
and let d1=(q1-p1)/|q1-p1|
.b2=p+a2d-pod
.q1
.p1
.b1=+a1d-pod
3. Start over with p1, d1
(more likely to be down the middle of the round cluster and therefore produce barrel gaps (linear and radial)) r1 d p
Alternatively, one could keep finding points on the sphere (like b1 and b2) until one has n-1 of them (n=dimension of space). Then there is a formula for the center of the circle through those points (at least in low dimensions???). However, even if there is a formula in high dimensions, it would be a nasty one and would take lots of rounds of the above to preduce the n-1 points (~ n/2 rounds). If n=17,000 for instance, that makes it infeasible.

5. .m2 .p' .m1 Oblique FAUST Pipe 2 d p
1. Linear project onto d-line from within a pipe.
If no good gaps, reset p, d
2.  linear gapped region (mean=m1), increase radius until there appears a region between gaps or thinnings
(at r1
and at r2).
Let m2=mean(2nd pre-thinning barrel stave).
.m2 r3 r4
Let r3=(r2+r1)/2, r4=r2 - r3
Reset d-line thru p'=(r3*m2+r4*m1)/2 r2
.p'
3. Look again for radial gap. if none goto1.
4. If radial gap, restrict to full barrel and look for linear gaps. If none goto 1.
6. If linear gap, declare subcluster, mask off and goto 1.
.m1 r1 d p

6. Q&A
f=distance dominated functional, avgGap=(fmax-fmin)/|f(X)|
may be a good measurement for setting thresholds, e.g.,
x is an outlier=anomaly if gap around {x} > 3*avgGap?
If the minimum barrel radii >> 0, we have chosen a d-line far from the data.
It may be advisable to pick p to ba an actual data point.
Here are the formulas from the spreadsheet:
G=(B12-B$6)*B$9+(C12-C$6)*C$9+(D12-D$6)*D$9+(E12-E$6)*E$9
H=G12-$G$ L=(x-p)od-min
I=(B12-B$6)^2+(C12-C$6)^2+(D12-D$6)^2+(E12-E$6)^2
B=SQRT[(x-p)o(x-p)-(x-p)od^2]
Note we don't round, so we are calculating pTree bitslices by truncating.
We don't even need to do that! For fixed piont, here are the bislice formulas:
Keep going (take bitslices to the right of decimal pt)
...
Floating point? Bitslice the mantissa. The exponent shifts the slice name. E.g.,
.1011  25
.0010  24
.1010  2-1
If d and t are trained over DocumentTerm (DT) Gradient(F)=G=(Gd, Gt).
Instead of a LineSearch using F(s)=f +sG, always use 2D-RectangleSearch, F(sd,st)=F(f + sd*Gd + st*Gt). Set F/sd =0 and F/st=0.
It may be a better approach to find dense cells (sphere, barrel, cone) then
fuse them, because it's difficult to position themaround clusters (due to
bumps, protrusion etc.) (Not true for outlier clusters (singleton\doubleton))
An Akg: Start with a line and a small radius barrel around it.
Find dense regions between 2 consecutive gaps in this pipe.
This should identify portion of a dense cluster. Lots of ways to go from there:
a. Use centroid of dense pipe piece as sphere|barrel center.
b. Move to a better centroid for that cluster by a gradient asc/desc process
c. In a "GA mutation" fashion, jump to a nearby centroid, governed by some fitness function (e.g., count in dense pipe piece). 24 1 23 22 1 21 1 20
2-1
2-2 1
2-3
2-4 1
SSPTS = set of all SPTSs (columns of reals); V = n-dim vector space. Code operations on SSPTS (both 1 level or multi-level): 22 2
5/16
SSPTS  SSPTS  SSPTS (Binary Algebraic Operations): including: +, -, /, RWP =Row_Wise_Product
10110.
10.
.01010
{SPTSk}k=1..n  SSPTS (Unary ops.Typically SPTSk=Vk) incl: SDv (Square Distance from a fixed vector, vV)
Gap analytic tools: L(x)=xod, S(x)=(x-p)o(x-p) and then from those, B(x)=S(x)-L2(x)
(If T is the minimum gap threshold, use T2 for S and B )
Oblique FAUST, Barrel (OFLB)
Alternate Lpqx, Bpqx to get a cluster dendogram (topdown).
Take p=1st_TR pt? d=vomavg
Defining Avg Density? AvD = count / k=1..dim(maxk-mink)? This is for choosing good Thresholds.
MinGapThres=Tb,AvD≡ b*(1/ AvD)1/dim b=adjustable param
If we're given a TrainingSet, TR, with K classes, is
avgk=1..Kvomk a better mediod than VoM?
Take p=MinCorner, q=MaxCorner of box circumscribing {VoMk}k=1..K better than not circ box of TR?
DPv (Dot Product with a fixed vector, vV)
ERa = FP's EinRings (n=1, rR) result masks rows s.t. row < a
SPTS  R includes AGa = YC's Aggregates and iceberg queies: count, sum, avg, max, min, median, rank_k, top_k, IceBergQueries.
SSPTS  SSPTS (Unary Operations) including: SPc=Scalar_Product (Multiply each SPTS row by same constant, c. Use const SPTS? all rows=c, then RWP. More efficient? w/o forming const SPTS? Use c's bit pattern c only? (subset of previous with n = |SSPTS|?)
Note, SSPTS includes SPTSs of all cardinalities (= depths = # of rows)
It seems best to code on SSPTS rather than on SSPTSn (card(SPTS)=n).
Of course, it is very important to know what the rows represent so as to
avoid nonsense results, however, why restrict the operations themselves?
When SPTS operands are of different depths, the result SPTS's depth = depth of the shallowest operand (operate from the top of each).

7. OF LB...LB Clustering on Concrete(STrength,ConcreteMix,WAter,FineAggregate, AGgregate). Assess STerror L<40M<60H
if 1st B radius>>0,
use p=min_radius_pt p
T=MGW=12
d=x-n=
CONCRETE
ST CM WA FA AG
(x-p)od/4 Ct Gp C
(x-p)od/4 gp3 C11
29 3
(x-p)od/4 gp C21
43 1
Br/4 gp3 C211
10 1
86 1 L1
Br/4 gp3 C0
65 1 L1 M1 L1
L1 M1
d=4 L1 M1
L2 M1 C0 L6
L3 M1 C211 L1
Br/4 gp3 C1
79 1 L2
L11 M3
L11 M C11 L1 L4 M1
(x-p)od/4 gp C23
51 1 M3 M2 L1
L1 M2 H1 C12
(x-p)od/4 gp3 C12
27 2 H1
Br/4 gp3 C231
35 1 L1 M1 H3 L1 H1 M2 H1 M1 H1 L1
L1 M1 C231
(x-p)od/4 gp C22
57 1
L20 M9 H4 C1
Br/ gp3 C241
...
61 1
Br/4 gp3 C2
94 2 L3
(x-p)od/4 g3 C411
21 5 L1 M1
M1 . H5 L1
L1 M1 H5 C2411
L9 M C21 M1
M2 . H1 H5
L4 M3 H1 C22
L2 M4 H3 C23 H5
Br/4 gp3 C251
25 1 M1 H1 M1
(x-p)od/4 gp3 C24
...
58 1 M1 L1
L2 M3 H16 C24
L2 M3 H4 C25
L1 M2 H16 C241 M1
(x-p)od/4 gp3 C25
58 1 M1 M2
M1 H3 C26
(x-p)od/4 gp3 C26
56 1
M1 ' H3 H2 M1
M1 H1C251 M2
(x-p)od/4 gp3 C27
56 1 H1
M3 H1 C27 M2 M H1
(x-p)od/4 gp3 C33
32 1 M H5
Br/4 ct gp3 C3
124 3
(x-p)od/4 gp3 C31
70 3
L18 M26 H28 C2 L1 M H3
M3 H3 C31
(x-p)od/4 gp3 C32
34 1
L1 M1 H4 C32
L1 M1 d=4 . H4 H1
c (Clust dendogram w/o purity)
M1 H5 C33 H1
c0 c1 c2 c3 c4 H2 H1
c11 c12
c21 c22 c23 c24 c25 c26 c27
c31 c32 c33
L2 M12 H 17 C3 M1 H4 M2
c211
c231
c241
c251 M1 M3
M3 H1 C4
c2411 H1
Br/4 ct gp3 C4
116 1 H1 M3

8. Oblique FAUST (OF) Clustering: Linear (default) OFL, Spherical OFS, Barrel OFB, Conical OFC)
Assume a real number table, T, converted to a PTreeSet.
GapUpper d p
GapLower a1 a2
Each method uses a real valued functional from X to R and all methods are completely data parallel (data can be distributed over a cluster, processed in parallel (dot product), then the partial results sent home to be added.
No gaps show on the red, blue
or green projection lines
Bpdx x
gapBarrel
Lp,d:XR: Lp,d(x)=(x-p)od Oblique FAUST Linear (OFL) clustering
(Enclose clusters between (n-1)-dimensional hyperplanar gaps)
Find a1<a2 such that =GapLower={x | a1<Lpd(x)<a1+T} and
=GapUpper={x | a2<Lpd(x)<a2+T} and C={x|a1+T<Lpd(x)<a2}
Bp,d(x)=(x-p)o(x-p)-((x-p)od)2 Oblique FAUST Barrel (OFB) (Enclose clusters with barrel gaps)
Search for GapLower>T, GapUpper>T and GapBarrel>T2 (BR≡Barrel_Radius) d p
Note: Bpd(x) = Sp(x) L2pd(x)
Note: C2pd(x) = L2pd(x) / Sp(x)
Cp,d(x)=(x-p)od / (x-p)o(x-p) Oblique FAUST Cone (OFC) (Enclose clusters with cone gaps) r p
Sp(x)=(x-p)o(x-p) Oblique FAUST Spherical (OFS) (Enclose clusters with spherical gaps)
Search Sp for spherical gap, {x | r2  Sp(x) < (r+T)2}= so that the interior of the r-sphere about p encloses a sub-cluster.

9. Oblique FAUST Pipe 0
Clustering on SEEDS(
4.5<F<7.5
R Ct gp
60 2
p=min q=max
ClsAreaLnkeAcoeLnk)
...
F<4.5
R Ct gp
4.5<F<7.5
R Ct gp
... MinRad too
high! reset p
xod Ct gp
10 3
pipe
Thinning
xod Ct gp in pipe
4 5
1 region! Look for radial gaps
Thinning
xod Ct gp in pipe
4 5
1 region! Look for radial gaps
R Ct gp
73 1
The alg only specifies
looking at the first region,
but it is interesting that other clusters are revealed
Next, Lin gap anal in r=26 barrel.
xod Ct gp
5 7 d p
Notes:
1. OFP0 may not work well if the pipe runs through the edge of a round cluster. The philosophy is the probabilistically, pipes are more likely to run through the center region of a round cluster since there is more of it. Next we try ways of adjusting p so that the pipe is more in the center of the subcluster.
2. I also tried Spherical when it appeared from the pipe analysis that we were at the center of a cluster. So far this didn't work out.
0. Always start with a pd-line linear gap analysis, then:
1. Find gapped regions in pipe: Project inside of pd-pipe (small radius).  full linear gapped region, analyze for an Initial Radial Gap (IRG).
2. If IRG, increase linear region width until cap gaps appear..
3. Mask off that cluster GOTO 1, using revise p,d at this point or if either 2, 3 have no gaps

10. Oblique FAUST Pipe 01 Clustering on SEEDS
p=nnnn q=xxxx
R Ct gp
7 1
r0=1.5 pipe
L Ct gp
4 5
No thinnings so it's just
one dense region
In middle of dense portion, [=3],
find radial falloff, r1
R Ct gp 73
1. Linearly project in a pdr0-pipe.
2. For every very dense region (take just the most dense middle portion? Find first radial density falloff at r1. If none GOTO 5.
3. Linearly project pdr1 barrel to determine the linear extent of that dense region. If it fails to show up. GOTO 5.
4. Mask off that cluster
5. Revise p and d and GOTO 1.