Update README.md
Browse files
README.md
CHANGED
|
@@ -124,14 +124,10 @@ model.save_pretrained("Meta-Llama-3.1-405B-Instruct-quantized.w4a16")
|
|
| 124 |
|
| 125 |
## Evaluation
|
| 126 |
|
| 127 |
-
The model was evaluated on
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
--model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16",dtype=auto,gpu_memory_utilization=0.4,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
|
| 132 |
-
--tasks openllm \
|
| 133 |
-
--batch_size auto
|
| 134 |
-
```
|
| 135 |
|
| 136 |
### Accuracy
|
| 137 |
|
|
|
|
| 124 |
|
| 125 |
## Evaluation
|
| 126 |
|
| 127 |
+
The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA.
|
| 128 |
+
Evaluation was conducted using the Neural Magic fork of [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct) (branch llama_3.1_instruct) and the [vLLM](https://docs.vllm.ai/en/stable/) engine.
|
| 129 |
+
This version of the lm-evaluation-harness includes versions of ARC-Challenge, GSM-8K, and MMLU that match the prompting style of [Meta-Llama-3.1-Instruct-evals](https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals).
|
| 130 |
+
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
### Accuracy
|
| 133 |
|