Update README.md
Browse files
README.md
CHANGED
|
@@ -28,7 +28,7 @@ license: llama3.1
|
|
| 28 |
- **Model Developers:** Neural Magic
|
| 29 |
|
| 30 |
Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
|
| 31 |
-
It achieves an average score of
|
| 32 |
|
| 33 |
### Model Optimizations
|
| 34 |
|
|
@@ -146,11 +146,11 @@ This version of the lm-evaluation-harness includes versions of ARC-Challenge, GS
|
|
| 146 |
<tr>
|
| 147 |
<td>MMLU (5-shot)
|
| 148 |
</td>
|
| 149 |
-
<td>
|
| 150 |
</td>
|
| 151 |
-
<td>
|
| 152 |
</td>
|
| 153 |
-
<td>
|
| 154 |
</td>
|
| 155 |
</tr>
|
| 156 |
<tr>
|
|
@@ -176,37 +176,41 @@ This version of the lm-evaluation-harness includes versions of ARC-Challenge, GS
|
|
| 176 |
<tr>
|
| 177 |
<td>Hellaswag (10-shot)
|
| 178 |
</td>
|
| 179 |
-
<td>
|
| 180 |
</td>
|
| 181 |
-
|
|
|
|
|
|
|
| 182 |
</td>
|
| 183 |
</tr>
|
| 184 |
<tr>
|
| 185 |
<td>Winogrande (5-shot)
|
| 186 |
</td>
|
| 187 |
-
<td>
|
|
|
|
|
|
|
| 188 |
</td>
|
| 189 |
-
<td>
|
| 190 |
</td>
|
| 191 |
</tr>
|
| 192 |
<tr>
|
| 193 |
<td>TruthfulQA (0-shot)
|
| 194 |
</td>
|
| 195 |
-
<td>
|
| 196 |
</td>
|
| 197 |
-
<td>
|
| 198 |
</td>
|
| 199 |
-
<td>
|
| 200 |
</td>
|
| 201 |
</tr>
|
| 202 |
<tr>
|
| 203 |
<td><strong>Average</strong>
|
| 204 |
</td>
|
| 205 |
-
<td><strong>
|
| 206 |
</td>
|
| 207 |
-
<td><strong>
|
| 208 |
</td>
|
| 209 |
-
<td><strong>
|
| 210 |
</td>
|
| 211 |
</tr>
|
| 212 |
</table>
|
|
|
|
| 28 |
- **Model Developers:** Neural Magic
|
| 29 |
|
| 30 |
Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
|
| 31 |
+
It achieves an average score of 86.01 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 86.63.
|
| 32 |
|
| 33 |
### Model Optimizations
|
| 34 |
|
|
|
|
| 146 |
<tr>
|
| 147 |
<td>MMLU (5-shot)
|
| 148 |
</td>
|
| 149 |
+
<td>86.25
|
| 150 |
</td>
|
| 151 |
+
<td>85.97
|
| 152 |
</td>
|
| 153 |
+
<td>99.67%
|
| 154 |
</td>
|
| 155 |
</tr>
|
| 156 |
<tr>
|
|
|
|
| 176 |
<tr>
|
| 177 |
<td>Hellaswag (10-shot)
|
| 178 |
</td>
|
| 179 |
+
<td>88.33
|
| 180 |
</td>
|
| 181 |
+
<td>88.16
|
| 182 |
+
</td>
|
| 183 |
+
<td>99.80%
|
| 184 |
</td>
|
| 185 |
</tr>
|
| 186 |
<tr>
|
| 187 |
<td>Winogrande (5-shot)
|
| 188 |
</td>
|
| 189 |
+
<td>87.21
|
| 190 |
+
</td>
|
| 191 |
+
<td>85.95
|
| 192 |
</td>
|
| 193 |
+
<td>98.55%
|
| 194 |
</td>
|
| 195 |
</tr>
|
| 196 |
<tr>
|
| 197 |
<td>TruthfulQA (0-shot)
|
| 198 |
</td>
|
| 199 |
+
<td>64.64
|
| 200 |
</td>
|
| 201 |
+
<td>64.75
|
| 202 |
</td>
|
| 203 |
+
<td>100.17%
|
| 204 |
</td>
|
| 205 |
</tr>
|
| 206 |
<tr>
|
| 207 |
<td><strong>Average</strong>
|
| 208 |
</td>
|
| 209 |
+
<td><strong>86.63</strong>
|
| 210 |
</td>
|
| 211 |
+
<td><strong>86.01</strong>
|
| 212 |
</td>
|
| 213 |
+
<td><strong>99.28%</strong>
|
| 214 |
</td>
|
| 215 |
</tr>
|
| 216 |
</table>
|