I’ve been unsuccessful in freezing lower pretrained BERT layers when training a classifier using Huggingface. I’m using AutoModelForSequenceClassification particularly, via code below, and I want to freeze the lower X layers (ex: lower 9 layers). Is this possible in HuggingFace, and if so what code …

Yes, in PyTorch freezing layers is quite easy. It can be done as follows: from transformers import AutoModelForSequenceClassification model = AutoModelForSequenceClassification.from_pretrained(“bert-base-cased”, num_labels=1) for name, param in model.named_parameters(): if name.startswith(".…

[image] nielsr: for name, param in model.named_parameters(): if name.startswith("..."): # choose whatever you like here param.requires_grad = False Thank you so much nielsr for the quick and useful reply. I believe I got this to work. So to verify, that can be written prior to “T…

To verify which layers are frozen, you can do: for name, param in model.named_parameters(): print(name, param.requires_grad)

Would just add to this, you probably want to freeze layer 0, and you don’t want to freeze 10, 11, 12 (if using 12 layers for example), so “bert.encoder.layer.1.” rather than “bert.encoder.layer.1” should avoid such things.

May I know for subsequent operations such as model.train and model.eval, does it change the the param.requires_grad that is specify above? Or do I have to do the above, everytime I change between training and eval mode?

Freeze Lower Layers with Auto Classification Model

🤗Transformers

Morphe May 25, 2023, 1:30am 7

According to this comment How to freeze layers using trainer? - #7 by sgugger it will not.

Topic		Replies	Views
Gradual Unfreezing support for Fine tuning models 🤗Transformers	3	4138	August 26, 2020
How to freeze some layers of BertModel Beginners	8	17936	August 25, 2022
How to freeze layers while fine-tuning? 🤗Transformers	2	1017	May 16, 2025
How to freeze layers using trainer? Beginners	11	33010	May 26, 2024
Gradual Layer Freezing with huggingface model 🤗Transformers	1	944	February 10, 2021

Freeze Lower Layers with Auto Classification Model

Related topics