1. LogisticRegression Regularization AI Lab 방 성 혁

2. Job Flow [ex2data.txt] Format [118 by 3] mapFeature function [ mapped_x ][ y ] Format [118 by 15] … [118 by 1] To make over-fit model [ mapped_x ] Format [118 by 15] … Get h(x) [Cost function with regularization term] Cost Optimize minimized Cost x2000 [result analysis]

3. Step #1. Import Library & Load data import tensorflow as tf import numpy as np xy = np.genfromtxt("ex2data.txt", unpack=True, dtype='float32') cf ) Case if unpack=True else if (unpack=False) First, tensorflow and numpy package should be imported Load “ex2data.txt”, col as row, data type = float ex2data.txt … unpack = True unpack = False [3 by 118] matrix [118 by 3] matrix

4. Step #2. mapFeature  Separate feature and real-data  To make over-fitted data, the defined function below will generate more complex features def mapFeature(x1, x2): degree = 4 out = np.ones(x1.shape) for i in range(1, degree+1): for j in range(0, i+1): out = np.c_[out, (x1**(i-j) * x2**j)] return out x = xy[0:-1] y = xy[-1] y = y.reshape(1,118) Shape of x : 2 by 118 Shape of y : 1 by 118 mapped_x = mapFeature(x[0], x[1]) Shape of mapped_x : 15 by 118 01… 010.. …………

5. Step #3. Declare placeholder & variable X = tf.placeholder(tf.float32) Y = tf.placeholder(tf.float32) #W = initial weight for data, len(xmap) = 15 #W = tf.Variable(tf.random_uniform([1,len(mapped_x)], -1, 1)) #rand_uniform var 1 by 15 W = tf.Variable(tf.zeros([1,len(mapped_x)]) + 0.1) place holder for mapped_x place holder for y Initial weight(theta) as 0.1 Cf) placeholder : feed actual values when Session runs

6. Step #4. Hypothesis h = tf.matmul(W, X) hypothesis = tf.sigmoid(h) Cf) sigmoid function 1 by 118 since (1 by 15)*(15 by 118) 1 by 118 the same dimension with h

7. Step #5. Cost Function & Regularization # Regularization term lambda_value = 0.1 length = len(np.transpose(mapped_x)) reg = (lambda_value/(2*length)) * tf.reduce_sum(W[0,1:]*W[0,1:]) cost = -tf.reduce_mean(Y*tf.log(hypothesis) + (1-Y)*tf.log(1-hypothesis)) + reg Cost Function of logistic regression with regularization term

8. Step #6. Tensorflow GradientDescentOptimizer  Tensorflow offers several machine learning libraries.  “GradientDescentOptimizer” is one of them.  Optimizer takes a parameter, learning rate “alpha”  Function “minimize” optimizes the cost from cost function a = tf.Variable(0.01) optimizer = tf.train.GradientDescentOptimizer(a) train = optimizer.minimize(cost)

9. Step #7. Cost Optimization  The optimization is done with 2000-iteration of gradient descent Cost Optimize minimized Cost x2000 with tf.Session() as sess: sess.run(tf.initialize_all_variables()) print "Iteration | cost" for i in xrange(2001): sess.run(train, feed_dict={X:mapped_x, Y:y}) if i % 200 == 0: print i,"|", sess.run(cost, feed_dict={X:mapped_x, Y:y}) 1)When using tensorflow variable, all defined variables must be initialized 2)Actual values are fed when Session runs

10. Step #8. Cost Comparison Cost variance with regularization cost = -tf.reduce_mean(Y*tf.log(hypothesis) + (1-Y)*tf.log(1-hypothesis)) + reg Cost variance without regularization cost = -tf.reduce_mean(Y*tf.log(hypothesis) + (1-Y)*tf.log(1-hypothesis)) Without reg Iteration | cost 0 | 0.731728 200 | 0.699464 400 | 0.681611 600 | 0.668645 800 | 0.657765 1000 | 0.648073 1200 | 0.639213 1400 | 0.631004 1600 | 0.62333 1800 | 0.616107 2000 | 0.609274 With reg Iteration | cost 0 | 0.731787 200 | 0.699457 400 | 0.681623 600 | 0.668714 800 | 0.657917 1000 | 0.64833 1200 | 0.639593 1400 | 0.631522 1600 | 0.623999 1800 | 0.616941 2000 | 0.610284 Conclusion 1) With training data, the cost with reg_term might be more higher than the other one. 2) With test data, the cost with reg_term would be more lower than the other one.