机器学习

1 监督式学习

分类算法 预测测试样本是否属于某些类型中的一个
回归算法 主要用于预测某种连续性的变量

2 无监督式学习

3 半监督式学习

4 增强式学习

文本分类

词汇文档矩阵（term-document matrix）
BOW（bag of word）

文本清理,获取标签，获取向量化

import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
import csv
import sklearn
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')
def preprocessing(text):
#     text = text.decode("utf-8")
    tokens = [word for sent in nltk.sent_tokenize(text) for word in nltk.sent_tokenize(sent)]
    stop = stopwords.words('english')
    tokens = [token for token in tokens if token not in stop]
    tokens = [word for word in tokens if len(word) >= 3]
    tokens = [word.lower() for word in tokens]
    lmtzr = WordNetLemmatizer()
    tokens = [lmtzr.lemmatize(word) for word in tokens]
    preprocessed_text = ' '.join(tokens)
    return preprocessed_text
#获取标签
smsdata = open('1')
sms_data = []
sms_labels = []
csv_reader =csv.reader(smsdata,delimiter='\t')
try:
    for line in csv_reader:
        sms_labels.append(line[0])
        sms_data.append(preprocessing(line[1]))
except:
    pass
#取样
import numpy as np
trainset_size =int(round(len(sms_data)*0.70))
x_train = np.array([''.join(el) for el in sms_data[0:trainset_size]])
y_train = np.array([el for el in sms_labels[0:trainset_size]])
x_test = np.array([''.join(el) for el in sms_data[trainset_size+1:len(sms_data)]])
y_test = np.array([el for el in sms_labels[trainset_size+1:len(sms_labels)]])
print(x_train)
print(y_train)
#向量化器向量化
from sklearn.feature_extraction.text import CountVectorizer
sms_list = []
sms_exp = []
for i in x_train:
    sms_list.append(i)
for line in sms_list:
    sms_exp.append(preprocessing(line))
vectorizer = CountVectorizer(min_df = 1)
X_exp = vectorizer.fit_transform(sms_exp)
print("||".join(vectorizer.get_feature_names()))
print(X_exp.toarray()[0])
#tf-idf向量化
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(min_df =2 ,ngram_range=(1,2),stop_words = 'english',strip_accents = 'unicode', norm ='12')
try:
    X_train = vectorizer.fit_transform(x_train)
    X_test = vectorizer.transform(x_test)
except:
    pass

文本分类之构建分类器

1 朴素贝叶斯

2 决策树

3 随即梯度下降

4 逻辑回归

5 支持向量机（SVM）

随机森林算法

文本聚类

1 k均值法

2 层次聚类法

文本中的主题建模

1 LDA（隐式狄利克雷分布）

2 LSI（潜在语义索引）

GENSIM