how to make optuna multi-objective study work (with allennlp)?

Hello, I followed an allennlp example in this repo here and it worked just fine. But when I tested optuna.multi_objective.create_study I had this error:

[W 2021-04-12 21:21:58,411] Trial 0 failed because of the following error: TypeError(“object of type ‘float’ has no len()”) Traceback (most recent call last): File “/xx/lib/python3.7/site-packages/optuna/_optimize.py”, line 217, in _run_trial value_or_values = func(trial) File “/xx/lib/python3.7/site-packages/optuna/multi_objective/study.py”, line 317, in mo_objective values = objective(mo_trial) File “objective.py”, line 165, in objective metrics = trainer.train() File “/xx/allennlp2.0/allennlp/allennlp/training/trainer.py”, line 930, in train metrics, epoch = self._try_train() File “/xx/allennlp2.0/allennlp/allennlp/training/trainer.py”, line 1052, in _try_train callback.on_epoch(self, metrics=metrics, epoch=epoch, is_primary=self._primary) File “/xx/lib/python3.7/site-packages/optuna/integration/allennlp.py”, line 496, in on_epoch self._trial.report(float(value), epoch) File “/xx/lib/python3.7/site-packages/optuna/multi_objective/trial.py”, line 133, in report if len(values) != self._n_objectives: TypeError: object of type ‘float’ has no len()

It seems to me that the problem comes from the validation_metric in trainer:

TARGET_METRIC = ["accuracy_doc", "accuracy_block", "accuracy_turn"]

trainer = GradientDescentTrainer(
        model=model,
        optimizer=optimizer,
        data_loader=train_data_loader,
        validation_data_loader=validation_data_loader,
        validation_metric=["+"+str(m) for m in TARGET_METRIC], #metrics will be summed to make the is_best decision
        patience=PATIENCE,  # `patience=None` since it could conflict with AllenNLPPruningCallback
        num_epochs=EPOCHS,
        cuda_device=CUDA_DEVICE,
        serialization_dir=serialization_dir,
        callbacks=[AllenNLPPruningCallback(trial, "validation_accuracy_doc")],
        )

In the doc of allennlp, validation_metric could be a string or a list of string. If it’s a list then they will be summed up. I am wondering if that’s why I got only one float value passed to values here:

File “/xx/lib/python3.7/site-packages/optuna/multi_objective/trial.py”, line 133, in report if len(values) != self._n_objectives: TypeError: object of type ‘float’ has no len()

Then I searched in allennlp for a trainer that doesn’t do the summing for validation_metrics but in vain. Maybe I wasn’t in the right path. Do you have any idea how to solve it? Thank you in advance!

[fsmosca]

@chuyuanli

Here is an example on multi-objective from the revised optuna example, no optuna pruner and some revisions on data size, epoch, trials and others. Two metrics are used maximize accuracy and minimize loss. patience is None as we have two validation metrics which are in opposite direction. At the end we try to determine which trial has the best accuracy (max) and the best loss (min).

"""
Optuna example that optimizes a classifier configuration for IMDB movie review dataset.
This script is based on the example of AllenTune (https://github.com/allenai/allentune).
In this example, we optimize the validation accuracy of sentiment classification using AllenNLP.
Since it is too time-consuming to use the entire dataset, we here use a small subset of it.
"""

import os
import random
import shutil
import sys

import allennlp
import allennlp.data
import allennlp.models
import allennlp.modules
import numpy
from packaging import version
import torch

import optuna
from optuna.integration import AllenNLPPruningCallback


sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from subsample_dataset_reader import SubsampleDatasetReader  # NOQA


DEVICE = -1  # If you want to use GPU, use DEVICE = 0.
N_TRAIN_DATA_SIZE = 2000
N_VALIDATION_DATA_SIZE = 200
MODEL_DIR = os.path.join(os.getcwd(), "result")
TARGET_METRIC = ["accuracy", "loss"]


def prepare_data():
    indexer = allennlp.data.token_indexers.SingleIdTokenIndexer(lowercase_tokens=True)
    tokenizer = allennlp.data.tokenizers.whitespace_tokenizer.WhitespaceTokenizer()

    reader = SubsampleDatasetReader(
        token_indexers={"tokens": indexer},
        tokenizer=tokenizer,
        train_data_size=N_TRAIN_DATA_SIZE,
        validation_data_size=N_VALIDATION_DATA_SIZE,
    )

    train_data_loader = allennlp.data.data_loaders.MultiProcessDataLoader(
        data_path="https://s3-us-west-2.amazonaws.com/allennlp/datasets/imdb/train.jsonl",
        reader=reader,
        batch_size=16,
    )
    validation_data_loader = allennlp.data.data_loaders.MultiProcessDataLoader(
        data_path="https://s3-us-west-2.amazonaws.com/allennlp/datasets/imdb/dev.jsonl",
        reader=reader,
        batch_size=64,
    )

    vocab = allennlp.data.Vocabulary.from_instances(train_data_loader.iter_instances())
    train_data_loader.index_with(vocab)
    validation_data_loader.index_with(vocab)
    return train_data_loader, validation_data_loader, vocab


def create_model(vocab, trial):
    dropout = trial.suggest_float("dropout", 0, 0.5)
    embedding_dim = trial.suggest_int("embedding_dim", 16, 128)
    output_dim = trial.suggest_int("output_dim", 32, 128)
    max_filter_size = trial.suggest_int("max_filter_size", 3, 4)
    num_filters = trial.suggest_int("num_filters", 32, 128)

    embedding = allennlp.modules.Embedding(
        embedding_dim=embedding_dim, trainable=True, vocab=vocab
    )

    encoder = allennlp.modules.seq2vec_encoders.CnnEncoder(
        ngram_filter_sizes=tuple(range(2, max_filter_size)),
        num_filters=num_filters,
        embedding_dim=embedding_dim,
        output_dim=output_dim,
    )

    embedder = allennlp.modules.text_field_embedders.BasicTextFieldEmbedder({"tokens": embedding})
    model = allennlp.models.BasicClassifier(
        text_field_embedder=embedder, seq2vec_encoder=encoder, dropout=dropout, vocab=vocab
    )

    return model


def objective(trial):
    train_data_loader, validation_data_loader, vocab = prepare_data()
    model = create_model(vocab, trial)

    if DEVICE > -1:
        model.to(torch.device("cuda:{}".format(DEVICE)))

    lr = trial.suggest_float("lr", 1e-3, 1e-1, log=True)
    optimizer = torch.optim.SGD(model.parameters(), lr=lr)

    serialization_dir = os.path.join(MODEL_DIR, "trial_{}".format(trial.number))
    trainer = allennlp.training.GradientDescentTrainer(
        model=model,
        optimizer=optimizer,
        data_loader=train_data_loader,
        validation_data_loader=validation_data_loader,
        patience=None,
        validation_metric=[f'+{TARGET_METRIC[0]}', f'-{TARGET_METRIC[1]}'],
        num_epochs=6,
        cuda_device=DEVICE,
        serialization_dir=serialization_dir,
    )
    metrics = trainer.train()
    return metrics["best_validation_" + TARGET_METRIC[0]], metrics["best_validation_" + TARGET_METRIC[1]]


if __name__ == "__main__":
    if version.parse(allennlp.__version__) < version.parse("2.0.0"):
        raise RuntimeError(
            "`allennlp>=2.0.0` is required for this example."
            " If you want to use `allennlp<2.0.0`, please install `optuna==2.5.0`"
            " and refer to the following example:"
            " https://github.com/optuna/optuna/blob/v2.5.0/examples/allennlp/allennlp_simple.py"
        )

    random.seed(41)
    torch.manual_seed(41)
    numpy.random.seed(41)

    study = optuna.multi_objective.create_study(directions=["maximize", "minimize"])
    study.optimize(objective, n_trials=6, timeout=600)

    print(f'Number of finished trials: {len(study.trials)}\n')

    # Get best trial based on val accuracy and loss.
    best_val_accuracy, best_val_loss = None, None
    best_val_accuracy_trial, best_val_loss_trial = None, None

    for t, n in enumerate(study.trials):
        print(f'trial: {t}, params: {n.params}, values: {n.values}')
        
        accuracy, loss = n.values[0], n.values[1]

        if best_val_accuracy is None or accuracy > best_val_accuracy:
            best_val_accuracy = accuracy
            best_val_accuracy_trial = t

        if best_val_loss is None or loss < best_val_loss:
            best_val_loss = loss
            best_val_loss_trial = t

    print(f'\nbest trial by accuracy: {best_val_accuracy_trial}')
    print(f'best trial by loss    : {best_val_loss_trial}\n')

    if best_val_accuracy_trial == best_val_loss_trial:
        print(f'best trial : {best_val_accuracy_trial}')

    shutil.rmtree(MODEL_DIR)

Result

Number of finished trials: 6

trial: 0, params: {'dropout': 0.031648361852979534, 'embedding_dim': 126, 'output_dim': 84, 'max_filter_size': 4, 'num_filters': 69, 'lr': 0.007721222000958557}, values: (0.515, 0.6921045482158661)
trial: 1, params: {'dropout': 0.4560765639308059, 'embedding_dim': 105, 'output_dim': 53, 'max_filter_size': 3, 'num_filters': 94, 'lr': 0.04718241253156262}, values: (0.515, 0.6886949092149734)
trial: 2, params: {'dropout': 0.40808044089764967, 'embedding_dim': 42, 'output_dim': 102, 'max_filter_size': 3, 'num_filters': 101, 'lr': 0.04760782807248917}, values: (0.515, 0.6887470483779907)
trial: 3, params: {'dropout': 0.3577354184246052, 'embedding_dim': 126, 'output_dim': 71, 'max_filter_size': 4, 'num_filters': 84, 'lr': 0.014888047451441754}, values: (0.515, 0.6912245452404022)
trial: 4, params: {'dropout': 0.19889669468070748, 'embedding_dim': 46, 'output_dim': 67, 'max_filter_size': 4, 'num_filters': 72, 'lr': 0.005208601987187918}, values: (0.515, 0.6924658715724945)
trial: 5, params: {'dropout': 0.3542561512425994, 'embedding_dim': 34, 'output_dim': 81, 'max_filter_size': 4, 'num_filters': 117, 'lr': 0.020332808812191672}, values: (0.515, 0.690562516450882)

best trial by accuracy: 0
best trial by loss    : 1

Since the accuracy is the same for all trials, we can say that the best trial is 1 because it has the best loss or minimum loss.

[himkt]

[note] You may able to run multi-objective optimization if you remove callbacks from your trainer.

Apart from a pruner, AllenNLPExecutor could be extended for multi-objective optimization. I think we can support multi-objective optimization by making AllenNLPExecutor.run() return a list of target metrics. If you are interested in extending AllenNLPExecutor, I’d love to review your PR!