Machine Learning support (ONNX-ML operators)

Overview

Machine Learning algorithms from Scikit-learn framework or XGBoost package are not directly supported. After training step, the algorithms should be converted in ONNX format to be deployed and imported. As for ONNX DL models, the same embedded C-inference API is generated (see “Embedded Inference Client ST Edge AI API” article). The skl2onnx utility is typically used to convert the models in ONNX format.

• https://xgboost.readthedocs.io/en/latest/index.html

• https://scikit-learn.org/

• https://onnx.ai/sklearn-onnx/

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from skl2onnx import to_onnx

X = np.array(..., dtype=np.float32)
Y = np.array(..., dtype=np.float32)

clf = RandomForestClassifier(random_state=0, n_estimators=4, max_depth=3)
clf.fit(X, Y)

exported_onnx = to_onnx(clf, np.array(X).astype(np.float32))
with open("random_forest.onnx", "wb") as f:
    f.write(exported_onnx.SerializeToString())

Note

Other ML framework with ONNX exporter can be used, but to be aware that the import of the ONNX-ML models in ST Edge AI Core has been mainly tested with Scikit-learn v0.23.1, skl2onnx v1.10.3 and XGBoost v1.5.1.

Supported ONNX-ML operators

Reference: https://github.com/onnx/onnx/blob/master/docs/Operators-ml.md

ai.onnx.ml	description
ai.onnx.ArrayFeatureExtractor	Select elements of the input tensor based on the indices passed.
ai.onnx.LabelEncoder	Maps each element in the input tensor to another value.
ai.onnx.LinearClassifier	Linear Classifier.
ai.onnx.LinearRegressor	Generalized linear regression evaluation.
ai.onnx.Normalizer	Normalize the input.
ai.onnx.SVMClassifier	Support Vector Machine classifier.
ai.onnx.SVMRegressor	Support Vector Machine regression prediction and one-class SVM anomaly detection.
ai.onnx.Scaler	Rescale input data, for example to standardize features by removing the mean and scaling to unit variance.
ai.onnx.TreeEnsembleClassifier	Tree Ensemble classifier. Returns the top class for each of N inputs.
ai.onnx.TreeEnsembleRegressor	Tree Ensemble regressor. Returns the regressed values for each input in N.
ai.onnx.ZipMap	Creates a map from the input and the attributes.

Supported scikit-learn algorithms

Reference: https://scikit-learn.org/

Scikit-learn is a complete framework for machine learning (ML) in Python which allows to build a large range of supervided or unsupervied ML algorithms: classifier, regressor, decision Trees, …

• sklearn.ensemble

  • Random forest classifier (sklearn.ensemble.RandomForestClassifier)

  • Random forest regressor (sklearn.ensemble.RandomForestRegressor)

  • Extra-trees classifier (sklearn.ensemble.ExtraTreesClassifier)

  • Extra-trees regressor (sklearn.ensemble.ExtraTreesRegressor)

  • Gradient Boosting for classification (sklearn.ensemble.GradientBoostingClassifier)

  • Gradient Boosting for regression (sklearn.ensemble.GradientBoostingRegressor)

  • Histogram-based Gradient Boosting Classification Tree (sklearn.ensemble.HistGradientBoostingClassifier)

  • Histogram-based Gradient Boosting Regression Tree (sklearn.ensemble.HistGradientBoostingRegressor)

  • Bagging classifier (sklearn.ensemble.BaggingClassifier)

  • Bagging regressor (sklearn.ensemble.BaggingRegressor)

  • Isolation Forest Algorithm (sklearn.ensemble.IsolationForest)

  • AdaBoost classifier (sklearn.ensemble.AdaBoostClassifier)

• sklearn.cluster

  • K-Means clustering. (sklearn.cluster.KMeans)

• sklearn.tree

  • Decision tree classifier (sklearn.tree.DecisionTreeClassifier)

  • Decision tree regressor (sklearn.tree.DecisionTreeRegressor)

• sklearn.svm

  • C-Support Vector Classification. (sklearn.svm.SVC)

  • Nu-Support Vector Classification (sklearn.svm.NuSVC)

  • Linear Support Vector Classification. (sklearn.svm.LinearSVC)

  • Unsupervised Outlier Detection. (sklearn.svm.OneClassSVM)

• sklearn.linear_model

  • Logistic Regression (aka logit, MaxEnt) classifier. (sklearn.linear_model.LogisticRegression)

  • Classifier using Ridge regression. (sklearn.linear_model.RidgeClassifier)

  • Ordinary least squares Linear Regression. (sklearn.linear_model.RidgeClassifier)

• sklearn.naive_bayes

  • Gaussian Naive Bayes algorithm for classification (sklearn.naive_bayes.GaussianNB)

  • Multinomial Naive Bayes algorithm for classification (sklearn.naive_bayes.MultinomialNB)

  • Complement Naive Bayes algorithm for classification (sklearn.naive_bayes.ComplementNB)

  • Bernoulli Naive Bayes algorithm for classification (sklearn.linear_model.LinearRegression)

• sklearn.neighbors

  • Neighborhood Components Analysis (sklearn.neighbors.NeighborhoodComponentsAnalysis)

• sklearn.discriminant_analysis

  • Linear Discriminant Analysis. (sklearn.discriminant_analysis.LinearDiscriminantAnalysis)

• sklearn.preprocessing

  • Generate polynomial and interaction features. (sklearn.preprocessing.PolynomialFeatures)

  • Normalize samples individually to unit norm. (sklearn.preprocessing.Normalizer)

  • Scale each feature by its maximum absolute value. (sklearn.preprocessing.MaxAbsScaler)

  • Transform features by scaling each feature to a given range. (sklearn.preprocessing.MinMaxScaler)

  • Standardize features by removing the mean and scaling to unit variance. (sklearn.preprocessing.StandardScaler)

  • Scale features using statistics that are robust to outliers. (sklearn.preprocessing.RobustScaler)

• sklearn.mixture

  • Gaussian Mixture. (sklearn.mixture.GaussianMixture)

  • Variational Bayesian estimation of a Gaussian mixture. (sklearn.mixture.BayesianGaussianMixture)

Random Forest example

Reference: http://onnx.ai/sklearn-onnx/auto_examples/plot_convert_model.html

from skl2onnx.common.data_types import FloatTensorType
from skl2onnx import convert_sklearn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y)
clr = RandomForestClassifier()
clr.fit(X_train, y_train)

initial_type = [('float_input', FloatTensorType([None, 4]))]
onx = convert_sklearn(clr, initial_types=initial_type,
                      target_opset=10)

with open("rf_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())