Paligemma2 onnx export KeyError: "Unknown task: image-text-to-text

🤗Optimum
1

Hi ,
I tried to convert the paligemma2 3B parameter model with 224 image resolution to onnx using optimum and got this error:

$optimum-cli export onnx --model google/paligemma-3b-pt-224 paligemma-3b-pt-224_onnx/

KeyError: “Unknown task: image-text-to-text. Possible values are: audio-classification for AutoModelForAudioClassification, audio-frame-classification for AutoModelForAudioFrameClassification, audio-xvector for AutoModelForAudioXVector, automatic-speech-recognition for (‘AutoModelForSpeechSeq2Seq’, ‘AutoModelForCTC’), depth-estimation for AutoModelForDepthEstimation, feature-extraction for AutoModel, fill-mask for AutoModelForMaskedLM, image-classification for AutoModelForImageClassification, image-segmentation for (‘AutoModelForImageSegmentation’, ‘AutoModelForSemanticSegmentation’, ‘AutoModelForInstanceSegmentation’, ‘AutoModelForUniversalSegmentation’), image-to-image for AutoModelForImageToImage, image-to-text for (‘AutoModelForVision2Seq’, ‘AutoModel’), mask-generation for AutoModel, masked-im for AutoModelForMaskedImageModeling, multiple-choice for AutoModelForMultipleChoice, object-detection for AutoModelForObjectDetection, question-answering for AutoModelForQuestionAnswering, reinforcement-learning for AutoModel, semantic-segmentation for AutoModelForSemanticSegmentation, text-to-audio for (‘AutoModelForTextToSpectrogram’, ‘AutoModelForTextToWaveform’), text-generation for AutoModelForCausalLM, text2text-generation for AutoModelForSeq2SeqLM, text-classification for AutoModelForSequenceClassification, token-classification for AutoModelForTokenClassification, visual-question-answering for AutoModelForVisualQuestionAnswering, zero-shot-image-classification for AutoModelForZeroShotImageClassification, zero-shot-object-detection for AutoModelForZeroShotObjectDetection”

Please help if you have any solution. Is “image-text-to-text” task is available in optimum? If yes, how to use it?
Or is there any alternative method to convert the model to onnx?

2

It seems that this can be avoided by explicitly specifying a task (to one of the supported tasks).

3

I tried specifying one of the existing task image-to-text. But that throws another error

$optimum-cli export onnx --model google/paligemma-3b-pt-224 --task image-to-text paligemma-3b-pt-224_onnx/

ValueError: Trying to export a paligemma model, that is a custom or unsupported architecture, but no custom onnx configuration was passed as custom_onnx_configs. Please refer to Export a model to ONNX with optimum.exporters.onnx for an example on how to export custom models. Please open an issue at GitHub ¡ Where software is built if you would like the model type paligemma to be supported natively in the ONNX export.

4

Of course, some of the newer models are not supported, but I found a converted version of Paligemma2. Maybe the github version of ONNX supports it.

The best way to find out is to ask the ONNX Community, who distribute it…

5

@John6666 Thanks