BERTIN The idea is to use the Spanish portion of mC4 (which roughly amounts for 1TB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggested here . 2. Language The model will be trained in Spanish (r…

Hey! Awesome project! I’m in :stuck_out_tongue:

I think you have already a full working team, but just in case you need more people, I’d be willing to help. I don’t know if I could be of much use, but the opportunity to contribute to a model called BERTIN is too good to let it go :grin:

Awesome to see so much interest here! We can also make multiple teams for the same project - so feel free to keep posting if you’re interested :slight_smile:

For this project we can have at least 2 teams!

I also think this is a very good effort. If I can be of any help please count me in.

Fantastic. I’d love to participate in one of the teams, if possible. Thanks!

What a good idea. Please, count me in if possible!!

Hello @versae & Team, I am interested to be a part of such an amazing project & team. I will try my best to contribute to the BERTIN model. It would be nice if we could discuss some more learning resources that would be useful for this project.I can work in any time zone that is comfortable for eve…

BERTIN: PreTrain RoBERTa-large from scratch in Spanish

Flax/JAX Projects

Pablogps June 24, 2021, 9:37am 10

This project sounds good! I’d like to help with either this one or the Spanish GPT-2, wherever makes more sense to add one person.

Topic		Replies	Views
Pretrain RoBERTa-large from scratch in Finnish Flax/JAX Projects	1	380	June 29, 2021
Pretrain RoBERTa-large from scratch in Swedish Flax/JAX Projects	2	1103	July 5, 2021
RobIt : PreTrain RoBERTa-base from scratch in Italian Flax/JAX Projects	4	522	June 29, 2021
PreTrain RoBERTa from scratch in Hindi Flax/JAX Projects	24	2164	December 10, 2021
PreTrain RoBERTa from scratch in Portuguese Flax/JAX Projects	16	2602	October 4, 2021

BERTIN: PreTrain RoBERTa-large from scratch in Spanish

Related topics