It seems to happen quite often.
opened 07:22AM - 27 Feb 24 UTC
closed 04:00PM - 03 Apr 24 UTC
bug
### System Info
```shell
Optimum Habana ==1.10.4
tokenizers==0.15.2
torch==2.… 2.1
transformers==4.37.2
```
### Information
- [X] The official example scripts
- [ ] My own modified scripts
### Tasks
- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)
### Reproduction
1) Clone this optimum habana; https://github.com/HabanaAI/optimum-habana-fork/tree/habana-main
2) pip install --upgrade-strategy eager optimum[habana]
3) git clone https://github.com/huggingface/optimum-habana
cd optimum-habana && git checkout v1.10.4
4) Run the example script in this folder:
/optimum-habana/examples/translation$ python3 run_translation.py \
--model_name_or_path t5-11b \
--do_train \
--do_eval \
--source_lang en \
--target_lang ro \
--source_prefix "translate English to Romanian: " \
--dataset_name wmt16 \
--dataset_config_name ro-en \
--output_dir /tmp/tst-translation \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--overwrite_output_dir \
--predict_with_generate \
--use_habana \
--use_lazy_mode \
--use_hpu_graphs_for_inference \
--gaudi_config_name Habana/t5 \
--ignore_pad_token_for_loss False \
--pad_to_max_length \
--save_strategy epoch \
--throughput_warmup_steps 3 \
--bf16
Traceback (most recent call last):
File "/home/test1/demo/optimum-habana/examples/translation/run_translation.py", line 51, in <module>
from optimum.habana import GaudiConfig, GaudiSeq2SeqTrainer, GaudiSeq2SeqTrainingArguments
File "/home/test1/demo/gaudi2-demo-env/lib/python3.10/site-packages/optimum/habana/__init__.py", line 29, in <module>
check_synapse_version()
File "/home/test1/demo/gaudi2-demo-env/lib/python3.10/site-packages/optimum/habana/utils.py", line 180, in check_synapse_version
habana_frameworks_version_number = get_habana_frameworks_version()
File "/home/test1/demo/gaudi2-demo-env/lib/python3.10/site-packages/optimum/habana/utils.py", line 218, in get_habana_frameworks_version
return version.parse(output.stdout.split("\n")[0].split()[-1])
IndexError: list index out of range
### Expected behavior
Traceback (most recent call last):
File "/home/test1/demo/optimum-habana/examples/translation/run_translation.py", line 51, in <module>
from optimum.habana import GaudiConfig, GaudiSeq2SeqTrainer, GaudiSeq2SeqTrainingArguments
File "/home/test1/demo/gaudi2-demo-env/lib/python3.10/site-packages/optimum/habana/__init__.py", line 29, in <module>
check_synapse_version()
File "/home/test1/demo/gaudi2-demo-env/lib/python3.10/site-packages/optimum/habana/utils.py", line 180, in check_synapse_version
habana_frameworks_version_number = get_habana_frameworks_version()
File "/home/test1/demo/gaudi2-demo-env/lib/python3.10/site-packages/optimum/habana/utils.py", line 218, in get_habana_frameworks_version
return version.parse(output.stdout.split("\n")[0].split()[-1])
IndexError: list index out of range
It seems like your environment is not correctly set up. How do you usually run your workflows on Gaudi? Inside a Docker container? Or have you installed the Habana software stack directly on your server?
regisss
February 10, 2025, 6:10pm
3
@DSNG1830 Can you show the output of pip show optimum-habana please?
