🦙⛰️ BigLlama-3.1-681B-Instruct

🦙✨ mlabonne/BigLlama-3.1-1T-Instruct

This is an experimental self-merge using meta-llama/Meta-Llama-3.1-405B-Instruct and created with mergekit.

This is the direct successor of Meta-Llama-3-120B-Instruct, a self-merge of Llama 3 70B that produced a decent 120B model for tasks like creative writing.

I tweaked the range of duplicated layers to hopefully make a sensible model. Use it at your own risk!

🔍 Applications

I recommend using this model for creative writing with the Llama 3 chat template.

⚡ Quantization

TBD.

🏆 Evaluation

TBD.

🧩 Configuration

This model was merged using the passthrough merge method. The following YAML configuration was used to produce this model:

slices:
- sources:
  - layer_range: [0, 42]
    model: meta-llama/Meta-Llama-3.1-405B-Instruct
- sources:
  - layer_range: [21, 63]
    model: meta-llama/Meta-Llama-3.1-405B-Instruct
- sources:
  - layer_range: [42, 84]
    model: meta-llama/Meta-Llama-3.1-405B-Instruct
- sources:
  - layer_range: [63, 105]
    model: meta-llama/Meta-Llama-3.1-405B-Instruct
- sources:
  - layer_range: [84, 126]
    model: meta-llama/Meta-Llama-3.1-405B-Instruct
merge_method: passthrough
dtype: bfloat16

Here is the code I've used to generate the config and calculate the number of layers/parameters after passthrough:

def generate_yaml_config(range_size, total_layers, nb_parameters):
    new_size = total_layers + total_layers - range_size
    new_param = (nb_parameters / total_layers) * new_size
    print(f"New size = {new_size} layers")
    print(f"New parameters = {new_param:.2f}B")
    yaml_str = "slices:\n"

    for i in range(0, round(total_layers - range_size + 1), range_size // 2):
        start = i
        end = min(start + range_size, total_layers)
        yaml_str += f"- sources:\n"
        yaml_str += f"  - layer_range: [{start}, {end}]\n"
        yaml_str += f"    model: meta-llama/Meta-Llama-3.1-405B-Instruct\n"

    yaml_str += "merge_method: passthrough\n"
    yaml_str += "dtype: bfloat16\n"

    print(yaml_str)

    return new_size, new_param

# Example usage
new_size, new_param = generate_yaml_config(42, 126, 410)
new_size, new_param = generate_yaml_config(105, new_size, new_param)