用于 Python 的端到端机器学习工具包 (MLToolkit/mltk)
项目描述
MLToolKit 项目
www.mltoolkit.org
当前版本：PyMLToolkit [v0.1.11]
MLToolKit (mltk) 是一个 Python 包，提供了一组用户友好的功能，以帮助在数据科学研究、教学或生产重点项目中构建端到端机器学习模型。
介绍
MLToolKit 支持机器学习应用程序开发过程的所有阶段。
安装
pip install pymltoolkit
如果安装因依赖问题而失败，请使用 --no-dependencies 执行上述命令
pip install pymltoolkit --no-dependencies
功能
- 数据提取（SQL、平面文件、二进制文件、图像等）
- 探索性数据分析（统计汇总、单变量分析、可视化分布等）
- 特征工程（支持数字、文本、日期/时间。图像数据支持将集成在 v0.1 的后续版本中）
- 模型构建（目前仅支持二元分类和回归）
- 超参数调整 [v0.2 开发中]
- 交叉验证（将在 v0.1 的后续版本中集成）
- 模型性能分析、解释预测（LIME 和 SHAP）以及模型之间的性能比较。
- 用于执行模型构建和评分任务的 JSON 输入脚本。
- 模型构建 UI [为 v0.2 开发中]
- ML 模型构建项目 [正在为 v0.2 开发]
- Auto ML（自动机器学习）[为 v0.2 开发]
- 模型 Deploymet 和 Serving [包括在内，将针对 v0.2 进行改进]
支持的机器学习算法/包
- 随机森林分类器：scikit-learn
- 逻辑回归：statsmodels
- 深度前馈神经网络 (DFF)：张量流
- 卷积神经网络 (CNN)：张量流
- 梯度提升：catboost、xgboost、lightgbm
- 线性回归：statsmodels
- RandomForestRegressor：scikit-learn
- ...更多模型将在未来的版本中添加...
用法
import mltk
警告：Python 变量、函数或类名称
Python 解释器有许多内置函数。编码时可以覆盖他们的定义，而不会从 Python 解释器发出任何警告。( https://docs.python.org/3/library/functions.html ) 因此，避免将这些名称作为变量、函数或类名。
| 腹肌 | 全部 | 任何 | ASCII | 垃圾桶 | 布尔 | 字节数组 | 字节 |
| 可调用的 | chr | 类方法 | 编译 | 复杂的 | 德拉特 | 听写 | 目录 |
| divmod | 枚举 | 评估 | 执行 | 筛选 | 漂浮 | 格式 | 冻结集 |
| 获取属性 | 全局变量 | 有属性 | 哈希 | 帮助 | 十六进制 | ID | 输入 |
| 整数 | 实例 | 是子类 | 迭代器 | 连 | 列表 | 当地人 | 地图 |
| 最大限度 | 记忆视图 | 分钟 | 下一个 | 目的 | 十月 | 打开 | 秩序 |
| 战俘 | 打印 | 财产 | 范围 | 代表 | 反转 | 圆形的 | 放 |
| 设置 | 片 | 排序的 | 静态方法 | 字符串 | 和 | 极好的 | 元组 |
| 类型 | 变量 | 压缩 | __进口__ |
如果您偶然覆盖了任何内置函数（例如列表），请执行以下操作以引入内置定义。
del(list)
同样，避免在 DataFrame 的列名中使用特殊字符和空格。执行以下操作以从列名中删除特殊字符。
Data = mltk.clean_column_names(Data, replace='')
MLToolkit 示例
数据加载和探索
import numpy as np
import pandas as pd
import mltk as mltk
Data = mltk.read_data_csv(file=r'C:\Projects\Data\incomedata.csv')
Data = mltk.clean_column_names(Data, replace='')
Data = mltk.add_identity_column(Data, id_label='ID', start=1, increment=1)
DataStats = mltk.data_description(Data)
数据预处理和特征工程
# Analyze Response Target
print(mltk.variable_frequency(DataFrame=Data, variable='income'))
# Set Target Variables
targetVariable = 'HighIncome'
targetCondition = "income=='>50K'" #For Binary Classification
Data=mltk.set_binary_target(Data, target_condition=targetCondition, target_variable=targetVariable)
print(mltk.variable_frequency(DataFrame=Data, variable=targetVariable))
Counts CountsFraction%
income
<=50K 24720 75.91904
>50K 7841 24.08096
TOTAL 32561 100.00000
# Flag Records to Exclude
excludeCondition="age < 18"
action = 'flag' # 'drop' #
excludeLabel = 'EXCLUDE'
Data=mltk.exclude_records(Data, exclude_ondition=excludeCondition, action=action, exclude_label=excludeLabel) # )#
# Get list of uniques values in categorical variables
categoryVariables = set({'sex', 'nativecountry', 'race', 'occupation', 'workclass', 'maritalstatus', 'relationship'})
print(mltk.category_lists(Data, list(categoryVariables)))
# Merge unique categorical values
category_merges = [{'variable':'maritalstatus', 'category_variable':'maritalstatus', 'group_value':'Married', 'values':["Married-civ-spouse", "Married-spouse-absent", "Married-AF-spouse"]}]
Data = mltk.merge_categories(Data, category_merges)
# Show Frequency distribution of categorical variable
sourceVariable='maritalstatus'
table = mltk.variable_frequency(Data, variable=sourceVariable, show_plot=False)
table.style.background_gradient(cmap='Greens').set_precision(3)
# Response Rate For Categorical Variables
mltk.variable_responses(Data, variables=categoryVariables, target_variable=targetVariable, show_output=False, show_plot=True)
获取数字单位列表
mltk.get_number_units()
变量操作
# General form
{
'type':'category'
'out_type':'cat',
'include':True,
'operation':'bucket',
'variables': {
'source':'age',
'destination': None # None for mult-variable operations, variable1 (for pair operations), variable1a (for pair sequence operation)
},
'parameters': {
'labels_str': ['0', '20', '30', '40', '50', '60', 'INF'],
'right_inclusive':True,
"default":'OTHER',
"null": 'NA'
}
}
List of Avaiable Transformation
|- Date/Numeric Transformations (transform)
| |- normalize
| |- datepart
| |- dateadd
| |- log
| |- exponent
| |- segment (piecewise functions)
|- String Transformation (str_transform)
| |- normalize
| |- strcount
| |- extract
|- Multi-variable Operations (operation_mult)
| |- expression
|- Sequence Order Check (seq_order)
| |- seqorder
|- Numeric/Date Comparison* (comparison)
| |- numdiff
| |- ratio
| |- datediff
| |- rowmin (pair)
| |- rowmax (pair)
|- String Comparison* (str_comparison)
| |- levenshtein
| |- jaccard
| |- ..more to add ..
|- Pair comparison
List of Avaiable Discrete Feature Transforms
|- Binary Variable (condition)
|- Numeric to Catergory (buckets)
|- Entity Grouping (dictionary)
|- Pair Equality/Existance (pair_equality)
|- Category Merge(category_merge)
# Transform numeric variable
rule_set = {
"operation":"normalize",
'variables': {
'source':'age',
'destination':'normalizedage'
},
"parameters":{"method":"zscore"}
}
Data, transformed_variable = mltk.create_transformed_variable_task(Data, rule_set, return_variable=True)
# Create Categorical Variables from continious variables
sourceVariable='age'
table = mltk.histogram(Data, sourceVariable, n_bins=10, orientation='vertical', density=True, show_plot=True)
print(table)
# Divide to categories
rule_set = {
'operation':'bucket',
'variables': {
'source':'age',
'destination':None
},
'parameters': {
'labels_str': ['0', '20', '30', '40', '50', '60', 'INF'],
'right_inclusive':True,
"default":'OTHER',
"null": 'NA'
}
}
Data, categoryVariable = mltk.create_categorical_variable_task(Data, rule_set, return_variable=True)
mltk.variable_response(DataFrame=Data, variable=categoryVariable, target_variable=targetVariable, show_plot=True)
Counts HighIncome CountsFraction% ResponseFraction% ResponseRate%
ageGRP
1_(0,20] 2410 2 7.40149 0.02551 0.08299
2_(20,30] 8162 680 25.06680 8.67236 8.33129
3_(30,40] 8546 2406 26.24612 30.68486 28.15352
4_(40,50] 6983 2655 21.44590 33.86048 38.02091
5_(50,60] 4128 1547 12.67774 19.72963 37.47578
6_(60,INF) 2332 551 7.16194 7.02716 23.62779
TOTAL 32561 7841 100.00000 100.00000 0.24081
# Create One Hot Encoded Variables
Data, featureVariables, targetVariable = mltk.to_one_hot_encode(Data, category_variables=categoryVariables, binary_variables=binaryVariables, target_variable=targetVariable)
Data[identifierColumns+featureVariables+[targetVariable]].sample(5).transpose()
相关性
correlation=mltk.correlation_matrix(Data, featureVariables+[targetVariable], target_variable=targetVariable, method='pearson', return_type='list', show_plot=False)
拆分训练，验证测试数据集
TrainDataset, ValidateDataset, TestDataset = mltk.train_validate_test_split(Data, ratios=(0.6,0.2,0.2))
建筑模型
identifierColumns = ['ID']
modelDataStats = mltk.data_description(TrainDataset)
sample_attributes = {
'SampleDescription':'Adult Census Income Dataset',
'NumClasses':2,
'ClassLabelsMap':{'<=50K':0, '>50K':1},
'DataFormat':'table',
'RecordIdentifiers':identifierColumns,
'ModelDataStats':modelDataStats
}
score_parameters = {
'Edges':[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'Percentiles':[0, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0],
'Threshold':0.5,
'Quantiles':10,
'TargetClass': '>50K',
'ScoreVariable':'Probability',
'ScoreLabel':'Score',
'QuantileLabel':'Quantile',
'PredictedLabel':'Predicted'
}
分类模型
模型属性
model_attributes = {
'ModelID': None,
'ModelType':'classification',# 'regression'
'EnumerationType': 'binary', # 'multi' 'mono' None
'ModelName': 'IncomeLevel',
'Version':'0.1',
'TrainingMethod': 'supervised'
}
逻辑回归
model_parameters = {
'MLAlgorithm':'LGR', # 'RF', # 'NN', # 'CATBST', (# 'CNN', # 'XGBST')
'MaxIterations':50
}
随机森林
model_parameters = {
'MLAlgorithm':'RF', # 'LGR', # 'NN', # 'CATBST', (# 'CNN', # 'XGBST')
'NTrees':500,
'MaxDepth':100,
'MinSamplesToSplit':10,
'Processors':2,
'Verbose':True
}
神经网络
# Setup Architecture
# Binary classification
SimpleDFF_architecture = {'layers': [
{'name': 'Dense1', 'class_name': 'Dense', 'position':'input', 'config':{'units': 512, 'activation':'relu', 'input_shape':(48,)}},
{'name': 'Dense2', 'class_name': 'Dense', 'position':'hidden', 'config':{'units': 512, 'activation':'relu', 'kernel_regularizer':{'l1':0.01}}},
{'name': 'Dropout1', 'class_name': 'Dropout', 'position':'hidden', 'config':{'rate':0.5, 'noise_shape':None, 'seed':None}},
{'name': 'Dense3', 'class_name': 'Dense', 'position':'output', 'config':{'units': 2, 'activation':'softmax'}}
]}
# Binary classification
LogisticRegressionNN_architecture = {'layers': [
{'name': 'Dense1', 'class_name': 'Dense', 'position':'input', 'config':{'units': 2, 'activation':'softmax', 'input_shape':(32,)}}
]}
# Multi Class classification
n_classes = 8
SimpleImageClassifier_architecture = {'layers': [
{'name':'Conv2D1', 'type':'Conv2D', 'position':'input', 'config':{'filters':32, 'kernel_size':(3,3), 'strides':None, 'padding':'same', 'activation':'relu', 'input_shape':(128, 128, 1), 'data_format':'channels_last'}},
{'name':'MaxPooling2D1', 'type':'MaxPooling2D', 'position':'hidden', 'config':{'pool_size':(2,2), 'strides':None, 'padding':'same', 'data_format':'channels_last'}},
{'name':'Conv2D2', 'type':'Conv2D', 'position':'hidden', 'config':{'filters': 64, 'kernel_size': (3,3), 'strides':None, 'padding':'same', 'activation':'relu', 'data_format':'channels_last'}},
{'name':'MaxPooling2D2', 'type':'MaxPooling2D', 'position':'hidden', 'config':{'pool_size':(2,2), 'strides':None, 'padding':'same', 'data_format':'channels_last'}},
{'name':'Dropout1', 'type':'Dropout', 'position':'hidden', 'config':{'rate':0.5, 'noise_shape':None, 'seed':None}},
{'name':'Flatten1', 'type':'Flatten', 'position':'hidden', 'config':{'data_format':'channels_last'}},
{'name': 'Dense1', 'type':'Dense', 'position':'output', 'config':{'units': 256, 'activation':'relu', 'kernel_regularizer':None}},
{'name':'Dropout2', 'type':'Dropout', 'position':'hidden', 'config':{'rate':0.5, 'noise_shape':None, 'seed':None}},
{'name': 'Dense2', 'type':'Dense', 'position':'output', 'config':{'units':n_classes, 'activation':'softmax'}}
]}
model_parameters = {
'MLAlgorithm':'NN',
'BatchSize':512,
'InputShape':InputShape,
'num_classes':2, #change accordingly
'Epochs':10,
'metrics':['accuracy'],
'architecture':SimpleDFF_architecture,
'Verbose':True
}
CatBoost
model_parameters = {
'MLAlgorithm':'CBST',
'NTrees': 500,
'MaxDepth':10,
'LearningRate':0.7,
'LossFunction':'Logloss',#crossEntropy
'EvalMatrics':'Accuracy',
'Imbalanced':False,
'TaskType':'GPU',
'Processors':2,
'UseBestModel':True,
'Verbose':True
}
XGBoost
model_parameters = {
'MLAlgorithm':'XGBST',
'NTrees': 500,
'MaxDepth':10,
'LearningRate':0.7,
'LossFunction':'binary:logistic',
'EvalMatrics':['auc', 'error'],
'Regularization': {'L1':0.0, 'L2' 1.0},
'SamplesRatioPerTree':0.8,
'FeaturesRatioPerTree':1.0,
'Processors':2,
'EarlyStopAttempts':5,
'Verbose':True
}
光GBM
model_parameters = {
'MLAlgorithm':'XGBST',
'NTrees': 500,
'MaxDepth':10,
'LearningRate':0.7,
'LossFunction':'binary:logistic',
'EvalMatrics':['auc', 'error'],
'Regularization': {'L1':0.0, 'L2' 1.0},
'SamplesRatioPerTree':0.8,
'FeaturesRatioPerTree':1.0,
'Processors':2,
'EarlyStopAttempts':5,
'Verbose':True
}
构建模型
XModel = mltk.build_ml_model(TrainDataset, ValidateDataset, TestDataset,
model_variables=modelVariables,
variable_setup = None,
target_variable=targetVariable,
model_attributes=model_attributes,
sample_attributes=sample_attributes,
model_parameters=model_parameters,
score_parameters=score_parameters,
return_model_object=True,
show_results=False,
show_plot=True
)
print(XModel.model_attributes['ModelID'])
print(XModel.model_interpretation['ModelSummary'])
print('ROC AUC: ', XModel.get_auc(curve='roc'))
print('PRC AUC: ', XModel.get_auc(curve='prc'))
print(XModel.model_evaluation['RobustnessTable'])
XModel.plot_eval_matrics(comparison=False)
minProbability maxProbability meanProbability BucketCount ResponseCount BucketFraction ResponseFraction BucketPrecision CumulativeBucketFraction CumulativeResponseFraction CumulativePrecision
Quantile
1 0.00000 0.00008 3.85729e-06 652 3 0.10011 0.00192 0.00460 1.00000 1.00000 0.23967
2 0.00008 0.00432 1.52655e-03 651 9 0.09995 0.00577 0.01382 0.89989 0.99808 0.26582
3 0.00435 0.02042 1.10941e-02 652 14 0.10011 0.00897 0.02147 0.79994 0.99231 0.29731
4 0.02049 0.05702 3.58648e-02 650 20 0.09980 0.01281 0.03077 0.69983 0.98334 0.33677
5 0.05711 0.12075 8.51409e-02 652 65 0.10011 0.04164 0.09969 0.60003 0.97053 0.38767
6 0.12086 0.20457 1.63366e-01 651 109 0.09995 0.06983 0.16743 0.49992 0.92889 0.44533
7 0.20469 0.31870