RandomizedSearchCV with pytorch-tabnet

It appears that the TabNetClassifier does not have a get_params method for hyperparameter estimation.

Is this reproducible your end?

Many thanks

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-33-03d6c8d15377> in <module>()
      4 
      5 start = time()
----> 6 randomSearch.fit(X_train, y_train)
      7 
      8 

1 frames
/usr/local/lib/python3.6/dist-packages/sklearn/base.py in clone(estimator, safe)
     65                             "it does not seem to be a scikit-learn estimator "
     66                             "as it does not implement a 'get_params' methods."
---> 67                             % (repr(estimator), type(estimator)))
     68     klass = estimator.__class__
     69     new_object_params = estimator.get_params(deep=False)

TypeError: Cannot clone object 'TabNetClassifier(n_d=32, n_a=32, n_steps=5,
                 lr=0.02, seed=0,
                 gamma=1.5, n_independent=2, n_shared=2,
                 cat_idxs=[],
                 cat_dims=[],
                 cat_emb_dim=1,
                 lambda_sparse=0.0001, momentum=0.3,
                 clip_value=2.0,
                 verbose=1, device_name="auto",
                 model_name="DreamQuarkTabNet", epsilon=1e-15,
                 optimizer_fn=<class 'torch.optim.adam.Adam'>,
                 scheduler_params={'gamma': 0.95, 'step_size': 20},
                 scheduler_fn=<class 'torch.optim.lr_scheduler.StepLR'>, saving_path="./")' (type <class 'pytorch_tabnet.tab_model.TabNetClassifier'>): it does not seem to be a scikit-learn estimator as it does not implement a 'get_params' methods.

[meechos]

@Catypad, compatibility with sklearn is not established at the level of using sklearn pipelines as in GridSearchCV out of the box. You may try ParameterGrid as in the pseudocode below:

from sklearn.model_selection import ParameterGrid

# Function that instantiates a tabnet model.
def create_tabnet(n_d=32, n_steps=5, lr=0.02, gamma=1.5, 
                  n_independent=2, n_shared=2, lambda_sparse=1e-4, 
                  momentum=0.3, clip_value=2.):
    
    model = TabNetClassifier(
        n_d=n_d, n_a=n_a, n_steps=n_steps,
        lr=lr,
        gamma=gamma, n_independent=n_independent, n_shared=n_shared,
        cat_idxs=cat_idxs,
        cat_dims=cat_dims,
        cat_emb_dim=cat_emb_dim,
        lambda_sparse=lambda_sparse, momentum=momentum, clip_value=clip_value,
        optimizer_fn=torch.optim.Adam,
        scheduler_params = {"gamma": 0.95,
                         "step_size": 20},
        scheduler_fn=torch.optim.lr_scheduler.StepLR, epsilon=1e-15, verbose = 0
    )
    return model
                  
# Generate the parameter grid.
param_grid = dict(n_d = [24, 32],
                  n_a = [24],
                  n_steps = [3, 4, 5],
                  lr = [0.01, 0.02],
                  gamma = [1, 1.5, 2],
                  lambda_sparse = [1e-2, 1e-3, 1e-4],
                  momentum = [0.3, 0.4, 0.5],
                  n_shared = [2],
                  n_independent = [2],
                  clip_value = [2.],     
)

grid = ParameterGrid(param_grid)

search_results = pd.DataFrame() 
for params in grid:
    params['n_a'] = params['n_d'] # n_a=n_d always per the paper
    tabnet = create_tabnet()
    tabnet.set_params(**params)
    tabnet.fit(X_train=X_train, y_train=y_train,
               X_valid=X_valid, y_valid=y_valid,
               max_epochs=num_epochs, patience=patience,
               batch_size=batch_size, virtual_batch_size=virtual_batch_size,
              )

    y_prob = tabnet.predict_proba(X_test)
    score = accuracy_score(y_test, y_prob)
    
    results = pd.DataFrame([params])
    results['score'] = np.round(score, 3)
    search_results = search_results.append(results)

Hope it helps!

[meechos]

I hope my feedback has been received as constructive, as it’s meant to be.

That said, I’ll do my best to add it, cheers!