Hi, I am trying to finetune T5 using the Trainer class. I understand that Trainer doesn’t work out-of-the-box for seq2seq tasks and saw @patrickvonplaten ’s https://github.com/huggingface/transformers/pull/5840 which extends the Trainer class to work for Bert2Bert. Is my understanding correct that t…

Hi @melody-ju Not sure what you mean by "T5 doesn’t work out-of-the-box for seq2seq tasks ". T5 is a seq2seq model and it does work for seq2seq tasks. You can use Trainer for seq2seq tasks as it is. Patrick’s PR extends it so that generative metrics can be calculated (ROUGE, BLUE etc), it should …

[image] valhalla: You can use Trainer for seq2seq tasks as it is. Patrick’s PR extends it so that generative metrics can be calculated (ROUGE, BLUE etc), it should be okay if you calculate them after training the training is finished. Oops sorry, I meant that Trainer does not work out of the …

It works :). It can not calculate generative metrics right now , but that’s being added in Seq2SeqTrainer.

Gotcha! Looking forward to using Seq2SeqTrainer. In the meantime I would like to calculate validation metrics during training but I don’t understand how to manipulate the output given by Trainer in EvalPrediction as the “prediction”, in order to retrieve the id’s of the generated prediction. What i…

To calculate generative metrics during training either clone Patrics branch or Seq2SeqTrainer PR branch. The default Trainer returns the output of the final LM head layer which is why the shape is batch_size * max_output_len * vocab_size. The above branches instead calls the generate method inside…

@melody-ju did you manage to compute metrics? I am stuck exactly at the same point as you. model_checkpoint = "distilroberta-base" model = AutoModelForMaskedLM.from_pretrained(model_checkpoint).to('cuda') def compute_custom_metric(eval_pred): print(eval_pred.predictions.shape) # ==> (3387, 32,…

@valhalla How to use this script for t5large? I have a machine with 4x4 P100 GPU’s. I am still fairly new to PL. I think from what all I have tried that t5 large is too large to fit on a single GPU so in order to fine-tune it. I need to implement Shared training to fine T5 large or there is somethi…

Using Trainer class with T5 - what is returned in EvalPrediction dict?

🤗Transformers

valhalla September 8, 2020, 11:38am 3

For more T5 related tips T5 Finetuning Tips

Topic		Replies	Views
Trainer class, compute_metrics and EvalPrediction 🤗Transformers	8	15017	October 3, 2025
Having trouble to configure trainer for T5 model evaluation 🤗Transformers	1	75	December 9, 2025
[Urgent] trainer.predict() and model.generate creates totally different predictions 🤗Transformers	4	7072	February 1, 2021
What does EvalPrediction.predictions contain exactly? Beginners	8	9179	August 3, 2023
Trainer.evaluate() with text generation Beginners	5	3670	December 31, 2021

Using Trainer class with T5 - what is returned in EvalPrediction dict?

Related topics