Logistic Regression, Binary Classification, Sigmoid

https://wooiljeong.github.io/ml/tf2_4/

youtu.be/PIjno6paszY

https://github.com/hunkim/DeepLearningZeroToAll/tree/master/tf2

 

위의 블로그 코드와 sung kim 교수님 강의 참고. 이번 예제는 tf gradient tape method를 사용해 구현.

(블로그는 tf gradient tape 사용, sung kim 교수님 강의는 tf 1.0이고 github는 keras model로 구현한 tf 2.0 코드)

 

 

Hypothesis: 1 / 1+e^-x

Loss Function: -log(H): y=1, -log(1-H): y=0    >>>    c(H, y) = -ylog(H) - (1-y)log(1-H)

Cost Function: cost(W) = ∑c(H, y) / m

Minimizing Cost: W' = W - ∇cost(W)

 

 

1. sigmoid / cross entropy code:

import numpy as np
import matplotlib.pyplot as plt

#sigmoid
x = np.arange(-6, 6, 0.1)
y = 1/(1+np.exp(-x))

plt.title('Sigmoid Function', size=15, weight='bold')
plt.plot(x,y,color='orange', label='Sigmoid')
plt.axhline(0.5, color='red', label='Decision Boundary=0.5')
plt.legend(loc='upper left')
plt.show()

#Cross Entropy
x = np.arange(0.01, 1, 0.01)
y1 = -np.log(x)
y0 = -np.log(1-x)

plt.title('Cross Entropy Function', size=15, weight='bold')
plt.plot(x, y1, color='dodgerblue', label='y=1')
plt.plot(x, y0, color='tomato', label='y=0')
plt.xlabel('H(x)')
plt.ylabel('Cost')
plt.legend(loc='upper center')
plt.show()

 

 

 

 

 

 

 

2. Logistic Regression code:

import tensorflow as tf
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# train data
x_train = np.array([[1.,1.],[1.,2.],[2.,1.],[3.,2.],[3.,3.],[2.,3.]],dtype=np.float32)
y_train = np.array([[0.],[0.],[0.],[1.],[1.],[1.]],dtype=np.float32)

# test data
x_test = np.array([[3.,0.],[4.,1.]],dtype=np.float32)
y_test = np.array([[0.],[1.]],dtype=np.float32)

# weights, bias
tf.random.set_seed(987) #랜덤 값 고정
W = tf.Variable(tf.random.normal([2, 1], mean=0.0)) #shape는 2 by 1 matrix
b = tf.Variable(tf.random.normal([1], mean=0.0))

print('Weights: \n', W.numpy(), '\n\nBias: \n', b.numpy())

# Learning Rate
learning_rate = 0.01 #gradient descent에서 step의 크기, 너무 크지도, 작지도 않게 설정

# Hypothesis and Prediction Function
def predict(X):
    f = tf.matmul(X, W) + b #행렬 곱 사용
    hypothesis = 1 / (1 + tf.exp(-f))
    return hypothesis

# Training
for i in range(2000+1):

    with tf.GradientTape() as tape: #미분 연산이 차례대로 기록

        hypothesis = predict(x_train)
        
        #cross entropy 합의 평균
        cost = tf.reduce_mean(-tf.reduce_sum(y_train * tf.math.log(hypothesis) + (1 - y_train) * tf.math.log(1 - hypothesis)))        
        
        #dcost_dW, dcost_db 미분 값
        W_grad, b_grad = tape.gradient(cost, [W, b])
		
        #최종 업데이트 값은 그레디언트에 learning rate를 곱해서 값을 감소(sub) 시키기
        W.assign_sub(learning_rate * W_grad)
        b.assign_sub(learning_rate * b_grad)

    if i % 400 == 0:
        print(">>> #%s \n Weights: \n%s \n Bias: \n%s \n cost: %s\n" % (i, W.numpy(), b.numpy(), cost.numpy()))

hypo = predict(x_test)
print("Prob: \n", hypo.numpy())
print("Result: \n", tf.cast(hypo > 0.5, dtype=tf.float32).numpy())

 

 

 

 

weight과 bias가 조정되면서 cost가 많이 내려갔다. test에 대한 분류 예측을 잘했다.

 

 

# 이진 분류는 sigmoid, 다중 분류는 softmax, 둘은 같은 cross entropy 사용

# sigmoid 결과는 확률이 아님, softmax 결과는 각각이 계층에 속할 확률, softmax의 결과 합은 항상 1

# deep NN에서는 vanishing gradient 문제 때문에 sigmoid보다 ReLU 더 좋음

 

 

참고:

https://www.tensorflow.org/tutorials/customization/autodiff?hl=ko