Upload folder using huggingface_hub

LICENSE

@@ -0,0 +1,31 @@
+The following applies to all files in this repository, unless otherwise noted:
+
+Copyright (c) 2025 Aleph Alpha Research GmbH. All rights reserved.
+
+This project is licensed under the terms of the Open Aleph License 1.0, available at
+https://github.com/Aleph-Alpha/.github/blob/main/oal.pdf
+
+---
+Excerpt from the license text:
+
+Subject to the terms and conditions of this License, the Licensor grants you a non-exclusive, worldwide,
+non-transferable, non-sublicensable, and royalty-free limited right to use, copy, modify, distribute, make
+otherwise publicly available, and reproduce the Works and Derivative Works under Licensor’s copyright,
+for any Non-Commercial and Non-Administrative purpose.
+You may not use, copy, modify, distribute, make otherwise publicly available, reproduce, or sublicense the
+Works or Derivative Works except as expressly provided under and in accordance with this License.
+Your rights granted under this License will automatically terminate if you fail to comply with any of the
+terms of this License.
+
+EXCEPT FOR DAMAGES CAUSED BY INTENT OR FRAUDULENTLY CONCEALED
+DEFECTS, AND EXCEPT FOR DAMAGES RESULTING FROM BREACH OF ANY
+WARRANTY OR GUARANTEE EXPRESSLY GIVEN BY LICENSOR IN THE OPEN ALEPH LICENSE,
+IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY
+DAMAGES ARISING OUT OF THE OPEN ALEPH LICENSE OR THE USE OF THE WORK. ANY
+MANDATORY STATUTORY LIABILITY UNDER APPLICABLE LAW REMAINS
+UNAFFECTED.
+
+EXCEPT AS EXPRESSLY STATED IN THIS LICENSE OR REQUIRED BY APPLICABLE
+LAW, THE WORKS ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+OF ANY KIND INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES REGARDING
+THE CONTENTS, ACCURACY, OR FITNESS FOR A PARTICULAR PURPOSE.

README.md

@@ -0,0 +1,444 @@
+---
+language:
+- en
+- de
+license: other
+thumbnail: https://huggingface.co/Aleph-Alpha/llama-3_1-70b-tfree-hat-sft/raw/main/source/aleph_alpha_logo_thumbnail.png
+license_name: open-aleph-license
+license_link: LICENSE
+tags:
+- Aleph Alpha Research
+- pytorch
+- Hirarchical Autoregressive Transformer
+- HAT
+model-index:
+- name: llama-3_1-70b-tfree-hat-sft
+  results: []
+---
+
+<div align="center">
+  <img src="source/aleph_alpha_logo.svg" width="60%" alt="Aleph Alpha Research Logo" />
+</div>
+ 
+<div align="center" style="line-height: 1;">
+  <a href="https://aleph-alpha.com/research/" target="_blank" style="margin: 2px;">
+    <img alt="Homepage" src="source/aleph_alpha_homepage_badge.svg" style="display: inline-block; vertical-align: middle;" />
+  </a>
+  <a href="https://huggingface.co/Aleph-Alpha" target="_blank" style="margin: 2px;">
+    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-AlephAlpha%20Research-e3ff00?color=e3ff00&amp;logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+ 
+<div align="center" style="line-height: 1;">
+  <a href="https://twitter.com/Aleph__Alpha" target="_blank" style="margin: 2px;">
+    <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-AlephAlpha_Research-white?logo=x&amp;logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+  <a href="https://huggingface.co/Aleph-Alpha/llama-3_1-70b-tfree-hat-sft/blob/main/LICENSE" style="margin: 2px;">
+    <img alt="License" src="https://img.shields.io/badge/License-Open Aleph License-white?&amp;color=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+
+<hr>
+
+# llama-3_1-70b-tfree-hat-sft
+<!-- markdownlint-disable first-line-h1 -->
+<!-- markdownlint-disable html -->
+<!-- markdownlint-disable no-duplicate-header -->
+
+This model card provides an overview of our **tokenizer-free llama-3.1-70b-tfree-hat model** based on Llama, a foundation model developed by Aleph Alpha Research* and publicly available under the Open Aleph License, a license explicitly allowing for non-commercial research and educational use.
+
+The model is based on the Llama 3.1 70B base model’s pre-trained backbone, replacing the Llama tokenizer with our Hierarchical Autoregressive Transformer (HAT) architecture which is described originally in our [paper](https://arxiv.org/abs/2501.10322). This novel architecture integrates character-level encoding and decoding with the word-level backbone, allowing for improved text compression (less sequence positions) and performance in the languages it has been trained on, and potentially higher robustness to prompt changes, as well as improved adaptability to new languages & domains via fine-tuning.
+
+The model was pre- and post-trained in English & German on carefully curated data in compliance with applicable EU and national regulations, including copyright and data privacy laws. The model has not been optimized for code generation and math and is thus not evaluated extensively on respective benchmarks.
+
+Please note that the realized inference speed strongly depends on the maturity of the inference implementation beyond the intrinsic text compression of any model. The current publicly available inference implementation is in a non-optimized state, hence any benchmark on speed must take account of that. We are releasing an optimized [vLLM-based inference solution](https://github.com/Aleph-Alpha/vllm) that is still under active development.
+
+You can find all model weights and their corresponding safetensors conversions at the following links:
+
+| Model Name | Link |
+| --- | --- |
+| `llama-3_1-70b-tfree-hat-sft` | [Link](https://huggingface.co/Aleph-Alpha/llama-3_1-70b-tfree-hat-sft) |
+
+# Model Access
+
+We provide access to our model through the channels listed below.
+
+- **HuggingFace**: The model’s weights are available on HuggingFace under the [Open Aleph License](https://github.com/Aleph-Alpha/.github/blob/main/oal.pdf), a license explicitly allowing for non-commercial research and educational use.
+
+We do not collect PII (personally identifiable information) for any of these channels. We do not log user inputs to the models. We do not train on user data.
+
+**Note**: The same models are made available to users regardless of their geographic location and their input language but subject to sanction regimes, technology export regulations, and other restrictions that may apply. The same offering is provided to all countries within and external to the European Union if no legal restrictions apply.
+
+# How to use
+
+## Inference
+
+We release a vLLM-based inference implementation adapted to our llama-3.1-70b-tfree-hat model [here](https://github.com/Aleph-Alpha/vllm). Please note that this inference implementation is still under active development.
+
+## Prompt formatting
+
+The prompt format used for this model is identical to the [Llama prompt format](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/). We highly recommend using it when prompting the models to ensure optimal performance for the supervised fine-tuned and direct-preference-optimized model versions. You can format your prompt in the recommended format by setting `add_llama_template=True` in the `model._prepare_input` method.
+
+# Evaluation
+
+**Performance**: Our T-Free model delivers strong performance in both English and German. For evaluation purposes, we compare our model with [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct). Respective benchmarks and results can be found in the tables below.
+
+**Efficiency**: Our tokenizer-free approach results in improved text compression, providing a foundation for improved efficiency in inference speed. We measure in terms of words processed across all languages and domains. We define the metric as **tokenizer fertility** or **bytes per sequence position**, where a higher value indicates better performance. Latency and throughput are currently out of scope for research-centric evaluations and will be addressed in the future. Currently, our evaluation framework automatically measures **bytes per sequence position** across datasets, allowing us to derive text compression scores and analyze variations across different dataset distributions. The end to end resulting efficiency is dependend on the inference implementation beyond the scope of the here provided inference implementation and reported compression scores.
+
+**Disclaimer**: The results presented below were generated using our internal inference implementation, not the inference module mentioned above.We plan to make source-available both our evaluation framework and a high-performance vLLM integration for this model in the coming weeks to ensure reproducibility. Our goal with this initial release is to provide the community with a straightforward codebase that demonstrates the architecture and supports basic inference capabilities.
+
+**Metric Glossary**
+
+`log_acc`: Average Accuracy Loglikelihood<br>
+`norm_log_acc`: Average Normalized Loglikelihood Accuracy<br>
+`comp_acc`: Average Completion Accuracy<br>
+`norm_prob_mass`: Average Probability Mass Normalized<br>
+`bleu`: Average BLEU Score<br>
+`rouge_gm`: Average ROUGE-Geometric-Mean<br>
+`F1`: Average F1<br>
+`CS`: Chatbot Style<br>
+`IF`: Instruction Following<br>
+`LC`: Language Consistency<br>
+`CI`: Concordance Index<br>
+`ES`: Exponential Similarity
+
+## SFT Benchmarks
+
+**MTBench winrates**
+
+English/German MTBench numbers are based on datasets created with [FastChat](https://github.com/LumiOpen/FastChat) for the corresponding models.
+
+|    |   **vs.** [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) **(Eng)**  |   **vs.** [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) **(Ger)**   |
+| --- | --- | --- |
+|   llama-3_1-70b-tfree-hat-sft   |  0.633  |  0.639  |
+
+Comparison on a broader set of evals:
+
+|   Group     |   Task     |   Metric Name    |   Num Fewshot     |   [llama-3_1-70b-tfree-hat-sft]((https://huggingface.co/Aleph-Alpha/llama-3_1-70b-tfree-hat-sft))     |   [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)     |   [llama-3_1-70b-tfree-hat-sft]((https://huggingface.co/Aleph-Alpha/llama-3_1-70b-tfree-hat-sft)) Compression     |   [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) Compression     |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+|   Knowledge     |   MMLU     |   `norm_log_acc`     |   5     |   0.773   |   0.818   |   5.818   |   4.922   |
+|   Knowledge     |   Full Text MMLU     |   `norm_log_acc`     |   5     |   0.786   |   0.830   |   5.850   |   5.109   |
+|   Knowledge     |   MMLU Pro     |   `norm_log_acc`     |   5     |   0.513   |   0.573   |   5.135   |   4.115   |
+|   Knowledge     |   GPQA     |   `log_acc`     |   0     |   0.360   |   0.545   |   5.260   |   3.845   |
+|   Knowledge     |   BBH     |   `norm_log_acc`     |   3     |   0.652   |   0.706   |   5.332   |   4.435   |
+|   Knowledge     |   OpenBookQA     |   `norm_log_acc`     |   10     |   0.526   |   0.556   |   7.101   |   7.092   |
+|   Knowledge     |   TriviaQA     |   `comp_acc`     |   5     |   0.582   |   0.757   |   6.924   |   6.018   |
+|   Knowledge     |   TruthfulQA     |   `norm_prob_mass`     |   6     |   0.176   |   0.191   |   6.575   |   5.702   |
+|   Reasoning     |   ARC Easy     |   `norm_log_acc`     |   25     |   0.920   |   0.911   |   7.018   |   6.426   |
+|   Reasoning     |   ARC Challenge     |   `norm_log_acc`     |   25     |   0.739   |   0.741   |   6.860   |   6.244   |
+|   Reasoning     |   Winogrande     |   `norm_log_acc`     |   5     |   0.749   |   0.697   |   6.856   |   6.615   |
+|   Reasoning     |   HellaSwag     |   `norm_log_acc`     |   10     |   0.809   |   0.665   |   5.980   |   5.300   |
+|   German     |   MMMLU     |   `norm_log_acc`     |   5     |   0.715   |   0.783   |   6.630   |   3.934   |
+|   German     |   [ARC Easy DE](https://huggingface.co/datasets/openGPT-X/arcx)     |   `norm_log_acc`     |   25     |   0.848   |   0.825   |   7.872   |   4.926   |
+|   German     |   [ARC Challenge DE](https://huggingface.co/datasets/openGPT-X/arcx)     |   `norm_log_acc`     |   25     |   0.669   |   0.653   |   7.798   |   4.878   |
+|   German     |   [Winogrande DE](https://huggingface.co/datasets/demelin/wino_x)     |   `norm_log_acc`     |   5     |   0.793   |   0.761   |   7.225   |   5.374   |
+|   German     |   [HellaSwag DE](https://huggingface.co/datasets/openGPT-X/hellaswagx)     |   `norm_log_acc`     |   10     |   0.727   |   0.707   |   6.971   |   4.150   |
+|   German     |   [TruthfulQA DE](https://huggingface.co/datasets/openGPT-X/truthfulqax)     |   `norm_prob_mass`     |   6     |   0.170   |   0.174   |   7.378   |   4.674   |
+|   German     |   [GSM8K DE](https://huggingface.co/datasets/openGPT-X/gsm8kx)     |   `comp_acc`     |   8     |   0.630   |   0.139   |   4.822   |   3.323   |
+|   German     |   WMT16     |   `bleu`     |   3     |   37.380   |   38.841   |   6.807   |   5.077   |
+|   German     |   WMT16 Instruct     |   `bleu`     |   3     |   37.614   |   37.912   |   6.862   |   5.145   |
+|   Instruction Following     |   Alpaca Eval     |   `CS`     |   0     |   0.363   |   0.168   |   7.984   |   4.746   |
+|   Instruction Following     |   Alpaca Eval     |   `IF`     |   0     |   0.945   |   0.961   |   7.984   |   4.746   |
+|   Instruction Following     |   Alpaca Eval     |   `LC`     |   0     |   0.994   |   0.993   |   7.984   |   4.746   |
+|   Long context     |   QuALITY     |   `log_acc`     |   0     |   0.488   |   0.459   |   4.867   |   4.302   |
+|   Long context     |   ZeroSCROLLS MuSiQue     |   `F1`     |   0     |   0.450   |   0.522   |   5.638   |   4.428   |
+|   Long context     |   ZeroSCROLLS SpaceDigest     |   `ES`     |   0     |   0.779   |   0.404   |   5.154   |   4.480   |
+|   Long context     |   ZeroSCROLLS SQuALITY     |   `rouge_gm`     |   0     |   0.170   |   0.159   |   4.994   |   4.243   |
+|   Safety     |   Winogender     |   `norm_log_acc`     |   5     |   0.679   |   0.843   |   6.875   |   6.701   |
+
+# Training Details
+
+## Model Architecture
+
+The model uses a hierarchical autoregressive architecture consisting of three components: encoder, backbone, and decoder together with connector layers between components. Encoder, backbone, and decoder are all instances of autoregressive transformers with pre-norm residual blocks in the style of Llama, using a SwiGLU unit as a feed-forward block, with all model parameters active during training and inference. The backbone model uses standard causal attention, while the encoder and decoder use local causal attention with a finite look-back window.
+
+The encoder processes input text as a sequence of UTF-8 bytes and produces a sequence of activations of the same length. This sequence is then split into chunks corresponding to words or other semantic units in the text (this is further explained below). In the encoder-backbone connector layer, for each word, a learned latent vector cross-attends to its corresponding chunk of encoder activations. The resulting sequence of latent vectors then serves as input to the backbone. The backbone processes this latent sequence and produces a sequence of word-level representations. Finally, the decoder module is another transformer that acts on the byte-level activations and has an LM head that produces next-byte probabilities. To make use of the higher level information stored in the word-level embeddings during decoding, another cross-attention mechanism is used. In each transformer block of the decoder, every byte-level position cross-attends to the backbone’s word-level representations that correspond to the words preceding this byte.
+
+## Encoder module
+
+|    |   **70B**   |
+| --- | --- |
+|   Number of layers   |   6   |
+|   Number of attention heads   |   16   |
+|   Head size   |   128   |
+|   Number of Key-Value heads   |   16   |
+|   Hidden size   |   2048   |
+|   Cross-attention hidden size   |   8192   |
+|   MLP expansion factor   |   2.75   |
+|   MLP type   |   SwiGLU   |
+|   Sequence length   |   98304   |
+|   Position embeddings   |   RoPE with base 1e5   |
+|   Attention type   |   causal, local with window size 768   |
+
+## Backbone module
+
+|    |   **70B**   |
+| --- | --- |
+|   Number of layers   |   80   |
+|   Number of attention heads   |   64   |
+|   Head size   |   128   |
+|   Number of Key-Value heads   |   8   |
+|   Hidden size   |   8192   |
+|   MLP expansion factor   |   3.5   |
+|   MLP type   |   SwiGLU   |
+|   Sequence length   |   12288   |
+|   Position embeddings   |   RoPE with base 5e5   |
+|   Attention type   |   causal   |
+
+## Decoder module
+
+|    |   **70B**   |
+| --- | --- |
+|   Number of layers   |   4   |
+|   Number of attention heads   |   16   |
+|   Head size   |   128   |
+|   Number of Key-Value heads   |   16   |
+|   Hidden size   |   2048   |
+|   Cross-attention hidden size   |   2048   |
+|   MLP expansion factor   |   2.75   |
+|   MLP type   |   SwiGLU   |
+|   Sequence length   |   98304   |
+|   Position embeddings   |   RoPE with base 1e5   |
+|   Attention type   |   causal, local with window size 768   |
+
+**Total parameter count**
+
+Total: `69,302,847,488` Encoder: `476,610,560` Backbone: `68,452,352,000` Decoder: `373,884,928`
+
+**Word splitter**
+
+To split arbitrary byte sequences, we adopted the guidelines from [UAX #29](https://unicode.org/reports/tr29/), which splits text into words for common Western languages but also produces meaningful semantic units for other types of languages (e.g. Chinese, Japanese, Korean). From now on, we refer to these splits as words.
+
+We also merged leading whitespace and trailing punctuation into the words to reduce sequence length at the word level.
+
+To improve the processing of code and math documents, we made additional adjustments to the Unicode splitter. First, we split instances of camel cases like FooBar into Foo and Bar. Second, we treated math symbols (again by Unicode standard) as separate words.
+
+## Pre-Training
+
+**Approach**
+
+We randomly initialized all model parameters of the encoder, decoder, and connector layers. The backbone architecture precisely matches the Llama 3.1 70B architecture, this allowed us to initialize the weights to the pre-trained Llama 3.1 70B base model weights. The model was then trained on the next-byte-prediction objective on a large and diverse document corpus (see below). Initially, we trained on sequences up to 3500 words with global batch size of 1024 for 30,000 steps, totaling to an amount of 108B words. We then continued training on sequences of up to 16000 words with global batch size 128 for another 5000 steps, totalling to another 10.2B words, where we upweight longer documents to make use of the extended context. The training was conducted in our [Scaling framework](https://github.com/Aleph-Alpha/scaling).
+
+**Data sources**
+
+The model was trained on a filtered subset of diverse corpora of text data including proprietary curated datasets, high-quality web content, public domain sources, German texts, mathematical texts, and programming code. The proportions and sources of data we used in the pre-training were:
+
+English Language Data (70%)
+
+- curated web and synthetic data (63%)
+
+- high quality curated sources such as Wikipedia and public domain books (7%)
+
+German Language Data (7%)
+
+- curated web and synthetic data (6.3%)
+
+- high quality curated sources such as Wikipedia and public domain books (0.7%)
+
+Mathematical Content (5%)
+
+- mathematical code and proofs (2%)
+
+- mathematical word problems and equations (3%)
+
+Programming Code (18%)
+
+- general programming code (11%)
+
+- high-quality and synthetic Python code (7%)
+
+## Data curation
+
+We applied a range of curation techniques, e.g., for German as described in [Aleph-Alpha-GermanWeb](https://huggingface.co/datasets/Aleph-Alpha/Aleph-Alpha-GermanWeb). These include but are not limited to: 
+
+- URL filtering. We used a URL filter developed to filter out fraudulent, harmful, and illegal content from an explicit blocklist, e.g., adult websites, or URLs containing words associated with fraudulent, harmful, or adult content.
+
+- Text extraction. Natural language texts which were embedded HTML and other web programming languages were extracted using the [Resiliparse](https://github.com/chatnoir-eu/chatnoir-resiliparse) text extractor.
+
+- Language identification. We used a [fastText language classifier](https://fasttext.cc/docs/en/language-identification.html) trained on character n-grams from Wikipedia to identify, retain, and sort texts into English and German.
+
+- Repetition removal. We applied heuristic methods for detection and removal of repetitions on the line, paragraph, and character level.
+
+- Document- and line-level filtering. We utilized additional document-level heuristics to ensure documents had reasonable numbers and quality of words, naturalistic symbols-to-words and numbers-to-words ratios, not predominantly made up of bullet points, and a sufficient quantity of real words.
+
+- Deduplication. Using exact and fuzzy deduplication to remove duplicate documents.
+
+## Synthetic data
+
+We also generated synthetic data by using permissively-licensed LLMs.
+
+## Instruction Fine-tuning 
+
+### Approach
+
+We optimized `llama-3_1-70b-tfree-hat-sft` for instruction-following using a standard post-training pipeline. We applied supervised fine-tuning (SFT) to train the model on both single-turn and multi-turn (chat) instruction-following tasks.
+
+### Data
+
+The data used for instruction fine-tuning is based on a mixture of user prompts and model competitions. The data mixture consists of roughly 2M samples from diverse datasets including but not limited to: specialized reasoning datasets covering mathematics, programming, and logical inference; human feedback focused on helpful and harmless responses; a small curated set for specific response patterns; safety and robustness subsets for appropriate boundaries; collaborative conversational data; multilingual conversation prompts; tabular data reasoning for structured information; and formal mathematics with advanced problems.
+
+We synthesized responses to the prompts using Qwen 2.5-32B and Qwen 2.5-72B. Additionally, we improved German performance by translating English prompts using Mistral-Nemo-Instruct-2407, generating the corresponding answers using Mistral-Small-3.1-Instruct, and performing quality filtering using an LLM judge based on Llama-3.3-70B-Instruct. Lastly, we supplemented the synthetic data with proprietary human-generated SFT data as well as further data sources.
+
+## Legal Compliance
+
+We acknowledge and abide by applicable national and international regulations, including copyright, data privacy, and other related legislation. Any text and data mining by us is performed in compliance with Directive (EU) 2019/790 and its respective national transposition. During the training and fine-tuning of our models, we comply with applicable data privacy laws, including Regulation (EU) 2016/679 (GDPR) and national data privacy regulations. To the extent possible and foreseeable, we also took legislation with forthcoming obligations into account, such as the obligations for General Purpose AI Models under Regulation (EU) 2024/1689 (EU AI Act), and will constantly monitor such developments and adapt our products and this model card accordingly.
+
+# Resource Usage
+
+## Compute & Training Efficiency 
+
+The following table shows the compute resources used in the training stages for the 70B model.
+
+|   **Model**   |   **Training phase**   |   **GPUs**   |   **Approximate average power consumption per GPU**   |   **Approximate GPU hours**   |
+| --- | --- | --- | --- | --- |
+|   70B   |   Continued pre-training   |   512 x H200   |   460W   |   49,920   |
+|   70B   |   Long context adaptation   |   256 x H100   |   160W   |  3200    |
+|   70B   |   Long context SFT   |   512 x H100   |   160W   |  12290  |
+
+## Environmental Impact
+
+Our H200 and A100 infrastructure runs entirely on 100% renewable energy, ensuring that no CO₂ emissions are directly incurred from training. In addition to this, the H200 data center boasts a power usage effectiveness (PUE) of ≤1.2. Its operation also maintains a net-zero water footprint. Specific number on renewable energy usage for the H100 GPUs is not yet available to us.
+
+To estimate the carbon footprint of inference, we base our calculations on publicly available data from the infrastructure provider and, where applicable, standard emissions accounting methodology. We report:
+
+- **Carbon emitted**: GPU runtime emissions
+
+- **Carbon emitted accounting for PUE**: GPU runtime emissions scaled by the data center's PUE
+
+Because the data centers operate fully on renewable energy, both metrics for its operation (excluding infrastructure-related emissions, e.g., initial chip manufacturing) are effectively zero. For H100 GPU infrastructure no information has been made available to us.
+
+|   Metric   |   H200 GPU   |   H100 GPU   |   A100 GPU   |
+| --- | --- | --- | --- |
+|   Carbon emitted   |   0 kg CO₂   |   no information available   |   0 kg CO₂   |
+|   Carbon emitted accounting for PUE   |   0 kg CO₂   |   no information available   |   0 kg CO₂   |
+
+## Power Consumption
+
+|   GPU Model   |   Max Power (W)   |
+| --- | --- |
+|   A100   |   400 W   |
+|   H100   |   700 W   |
+|   H200   |   700 W   |
+
+Numbers may be contextualized with reference to publicly available studies, such as the carbon footprint of language model training.
+
+# Intended Use
+
+These models are intended to be deployed as components of AI systems or applications. Use-cases and the model's capabilities include but are not limited to: text generation, classification, summarization, question answering, and labeling. Note that applications might require additional model adaptations or components for guarding against unwanted application behavior or model output.
+
+## Non-Permitted Use
+
+Our models shall not be used for illegal or unlawful actions of any kind and with any illegal or unlawful content. This includes in particular prohibited practices according to Article 5 of Regulation (EU) 2024/1689 (EU AI Act) and other activities such as engaging in terrorism, violence, human trafficking, illegal distribution of materials to minors, sexual solicitation, any other criminal activities, harassment, discrimination, creating or promoting malicious code or activities risking death or harm, including those related to military or nuclear applications, and activities not in compliance with sanction regimes, technology export regulations, and other restrictions that may apply. The models are to be used following ethical standards. The utilization of our technology is always governed by, and may be limited in accordance with, our Terms and Conditions, the Open Aleph License, or any specific agreement we might have established with you.
+
+Although we do not inspect the requests sent to our API, we regularly review and monitor potential violations that may be related to our models and depending on the circumstances of the specific case take legal action against them. This includes but is not limited to, enforcement to remove published model content, requesting compensation for damages caused, and account termination or removal of credits.
+
+For non-anonymous reports, we also provide an appeals mechanism for usage policy violations via our dedicated contact address [[email protected]](mailto:[email protected]) to communicate with us.
+
+Customers and partners are enabled to use our [ticketing system](https://servicedesk.aleph-alpha.de/external) for appeals, claims, and feedback.
+
+
+# Risks and Limitations
+
+**Note:** Language models are **not agents** and not optimized for prescriptive actions. The use of language models in high-stake environments, for critical decisions or to support a user's wellbeing should be performed with additional guardrails in place.
+
+## Risk Categories
+
+In the following sections, we describe risk categories and provide examples of completions we would consider inappropriate or harmful. We then describe steps to minimize these risks.
+
+**Harmful Language**
+
+Large language models can sometimes generate undesired outputs that are unsuitable for certain applications. This includes producing content with harmful language, discriminative content, inappropriate tone and style, systemic biases, or suggestions that might encourage illegal actions. Such outputs can also include incorrect, outdated information, or material that is not suitable for all ages. While we constantly take efforts to reduce the likelihood of such undesired outputs, this possibility can never be fully ruled out. To minimize these issues, the following strategies can be employed:
+
+- Abide by the guidance on illegal use provided for in this Model Card.
+    
+- Crafting prompts carefully to guide the model's output more effectively.
+    
+- Utilizing a finetuned model (often referred to as a control or instruct model) that prioritizes using explicitly provided information.
+    
+- Employing a finetuned model designed to maintain an appropriate tone and style, including avoiding offensive language.
+    
+- Conducting additional validations at the application level to ensure output quality and appropriateness.
+    
+
+### Systemic Biases
+
+Language models obtain world-knowledge from their pre-training data and may therefore exhibit the same systematic biases that are present in the data. Differing deployment scenarios (including differing cultural contexts) can expose systematic biases in different ways. We acknowledge the cultural diversity of communities and users inside and outside the EU. For larger deployments, we encourage users to track systematic biases relevant to their use-case, and we are happy to consult on bespoke fine-tunings to alleviate such biases.
+
+### Outdated World Knowledge
+
+Pre-training was performed using a fixed dataset, created at a fixed date in the past. Accordingly, the world knowledge of foundation models is limited to the information contained in its training data. More recent information may not be known to the model or misunderstood when presented as input during live usage. 
+
+Risks include:
+
+- Generation of personally identifiable information. Models are not explicitly trained to provide such information, but may seem to provide personally identifiable information. This does not necessarily imply the presence of such information in training data, as hallucination is possible.
+    
+- Generation of unintended, irrelevant, or repetitive outputs. This includes the production of incorrect or outdated information.
+    
+
+Risks may be mitigated by:
+
+- Injecting context, where relevant.
+    
+- Crafting prompts carefully to guide the model's output more effectively.
+    
+- Performing validations on the application layer, e.g., classifying the output.
+    
+- Using the repetition penalty, especially in the case of repetition, or other parameters available in the API (see [documentation](https://docs.aleph-alpha.com/api/complete/)).
+    
+- Avoiding of use cases targeted at retrieval of personally identifiable information.
+    
+
+### Political Bias
+
+Our models have not been optimized to represent a political opinion or take a specific point of view. They may generate outputs that contradict a user's opinion or expectation, e.g., produce hateful, violent or inappropriate, biased, or discriminatory content. Such behavior may be addressed by:
+
+- Crafting prompts carefully to guide the model's output more effectively.
+    
+- Performing validations on the application layer, e.g., via Red-Teaming or classifying the output.
+    
+
+### Mistaken for a Human
+
+Users may attribute human traits to AI models. This also includes the fact that content generated by the model is not explicitly detectable at this point. It is therefore required to:
+
+- Inform end users that they are interacting with or reading output of an AI.
+    
+- Design the system in a way that mitigates the impact of unintended interpretation of the output.
+    
+
+### Other Errors
+
+Any AI module can produce errors, even after implementing all the recommended measures. When integrating foundation language models into an application, users should:
+
+- be aware of the risk of (harmful) failure cases and implement the use case in a way that mitigates such risks.
+    
+- be aware that foundation models do not contain application logic, e.g., content filters. Enforcement policies relevant to the use case need to be implemented in the application layer.
+    
+- avoid unsupervised use in high-stakes environments.
+    
+- validate output with adequate measures.
+    
+
+### Mitigation Approach
+
+We specifically tailor model alignment and risk mitigation techniques to each user-facing application built on top of our models, working closely with our customers to refine them according to their unique requirements. Our intention is for these models to undergo further fine-tuning by us and our customers, utilizing their own datasets alongside our support and datasets to ensure suitability for end-user applications, including harm mitigation efforts. Our customers are responsible for adhering to the terms and conditions when aligning the models in their downstream applications.
+
+### Reproducibility
+
+Some inference parameters, e.g., temperature, lead to the random sampling of outputs, which precludes the reproducibility of outputs. Even when such parameters are not in use, outputs may diverge slightly on a numeric level for technical reasons. One may implement the following measures if needed:
+
+- Logging of past model outputs on the application layer (Aleph Alpha Research is not storing any data and/or using any data provided in prompts for the training of its LLMs).
+    
+
+This list of risks, biases, and limitations may not be complete, as improving the understanding and behavior of language models is an ongoing research topic in the AI science community.
+
+# Legal Acknowledgements
+
+- **Built with Llama**: Built with Llama: Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. The applicable license agreement can be found under the following link: [Llama 3.1 Community License Agreement ](https://www.llama.com/llama3_1/license/)
+
+- **Improved using Qwen**
+
+
+\*Aleph Alpha Research refers to Aleph Alpha Research GmbH
+
+[hat-paper]: https://arxiv.org/abs/2501.10322

config.json

@@ -0,0 +1,109 @@
+{
+  "architectures": [
+    "HATForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "config.HATArchitectureConfig",
+    "AutoModelForCausalLM": "model.HATForCausalLM"
+  },
+  "backbone_config": {
+    "hidden_size": 8192,
+    "intermediate_size": 28672,
+    "is_neox_style": true,
+    "key_query_norm": false,
+    "key_query_norm_per_head": false,
+    "max_position_embeddings": 12288,
+    "mlp_bias": false,
+    "num_attention_heads": 64,
+    "num_hidden_layers": 80,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "factor": 8.0,
+      "high_freq_factor": 4.0,
+      "low_freq_factor": 1.0,
+      "original_max_position_embeddings": 8192,
+      "rope_type": "llama3"
+    },
+    "rope_theta": 500000,
+    "sliding_window": null,
+    "transformers_version": null,
+    "use_cache": true,
+    "vocab_size": 0
+  },
+  "decoder_config": {
+    "cross_attention_config": {
+      "attention_num_kv_heads": 16,
+      "hidden_size": 2048,
+      "hidden_size_kv": 8192,
+      "hidden_size_q": 2048,
+      "key_query_norm": false,
+      "key_query_norm_per_head": false,
+      "num_attention_heads": 16,
+      "word_window_size": 1
+    },
+    "cross_attn_every_layer": true,
+    "hidden_size": 2048,
+    "intermediate_size": 5632,
+    "is_neox_style": true,
+    "key_query_norm": false,
+    "key_query_norm_per_head": false,
+    "max_position_embeddings": 98304,
+    "mlp_bias": false,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 4,
+    "num_key_value_heads": 16,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "rope_type": "default"
+    },
+    "rope_theta": 100000,
+    "sliding_window": 768,
+    "transformers_version": null,
+    "use_cache": true,
+    "vocab_size": 256
+  },
+  "encoder_config": {
+    "cross_attention_config": {
+      "attention_num_kv_heads": 64,
+      "hidden_size": 8192,
+      "hidden_size_kv": 2048,
+      "hidden_size_q": 8192,
+      "key_query_norm": false,
+      "key_query_norm_per_head": false,
+      "num_attention_heads": 64,
+      "word_window_size": 1
+    },
+    "hidden_size": 2048,
+    "intermediate_size": 5632,
+    "is_neox_style": false,
+    "key_query_norm": false,
+    "key_query_norm_per_head": false,
+    "max_position_embeddings": 98304,
+    "mlp_bias": false,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 6,
+    "num_key_value_heads": 16,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "rope_type": "default"
+    },
+    "rope_theta": 100000,
+    "sliding_window": 768,
+    "transformers_version": null,
+    "use_cache": true,
+    "vocab_size": 256
+  },
+  "max_position_embeddings": 98304,
+  "max_word_size": 100,
+  "model_type": "hierarchical_autoregressive_transformer",
+  "sliding_window": 768,
+  "special_token_dict": {
+    "<|begin_of_text|>": 250,
+    "<|end_header_id|>": 252,
+    "<|eot_id|>": 192,
+    "<|start_header_id|>": 251
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.3"
+}

config.py

@@ -0,0 +1,262 @@
+from dataclasses import dataclass
+
+import torch.nn as nn
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.llama.configuration_llama import LlamaConfig
+
+
+@dataclass
+class TransformerHATModelConfig(LlamaConfig):
+    def __init__(
+        self,
+        hidden_size: int,
+        num_hidden_layers: int,
+        num_attention_heads: int,
+        num_key_value_heads: int,
+        rms_norm_eps: float,
+        intermediate_size: int,
+        max_position_embeddings: int,
+        rope_scaling: dict,
+        rope_theta: float,
+        mlp_bias: bool,
+        use_cache: bool = True,
+        sliding_window: int | None = None,
+        vocab_size: int = 0,
+        hidden_act: str = "silu",
+        key_query_norm: bool = False,
+        key_query_norm_per_head: bool = False,
+        is_neox_style: bool = True,
+        **kwargs,
+    ):
+        super().__init__(
+            vocab_size=vocab_size,
+            hidden_size=hidden_size,
+            num_hidden_layers=num_hidden_layers,
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            hidden_act=hidden_act,
+            rms_norm_eps=rms_norm_eps,
+            intermediate_size=intermediate_size,
+            max_position_embeddings=max_position_embeddings,
+            rope_scaling=rope_scaling,
+            rope_theta=rope_theta,
+            mlp_bias=mlp_bias,
+            use_cache=use_cache,
+            **kwargs,
+        )
+
+        self.sliding_window = sliding_window
+        self.key_query_norm = key_query_norm
+        self.key_query_norm_per_head = key_query_norm_per_head
+        self.is_neox_style = is_neox_style
+
+    def to_dict(self):
+        config_dict = {
+            "vocab_size": self.vocab_size,
+            "hidden_size": self.hidden_size,
+            "num_hidden_layers": self.num_hidden_layers,
+            "num_attention_heads": self.num_attention_heads,
+            "num_key_value_heads": self.num_key_value_heads,
+            "rms_norm_eps": self.rms_norm_eps,
+            "intermediate_size": self.intermediate_size,
+            "max_position_embeddings": self.max_position_embeddings,
+            "rope_scaling": self.rope_scaling,
+            "rope_theta": self.rope_theta,
+            "mlp_bias": self.mlp_bias,
+            "use_cache": self.use_cache,
+            "sliding_window": self.sliding_window,
+            "transformers_version": self.transformers_version,
+            "key_query_norm": self.key_query_norm,
+            "key_query_norm_per_head": self.key_query_norm_per_head,
+            "is_neox_style": self.is_neox_style,
+        }
+        return config_dict
+
+
+@dataclass
+class CrossAttentionConfig:
+    def __init__(
+        self,
+        hidden_size: int,
+        hidden_size_q: int,
+        hidden_size_kv: int,
+        num_attention_heads: int,
+        attention_num_kv_heads: int,
+        word_window_size: int,
+        key_query_norm: bool, 
+        key_query_norm_per_head: bool,
+    ):
+        self.hidden_size = hidden_size
+        self.hidden_size_q = hidden_size_q
+        self.hidden_size_kv = hidden_size_kv
+        self.num_attention_heads = num_attention_heads
+        self.attention_num_kv_heads = attention_num_kv_heads
+        self.word_window_size = word_window_size
+        self.key_query_norm = key_query_norm
+        self.key_query_norm_per_head = key_query_norm_per_head
+
+    def to_dict(self):
+        return {
+            "hidden_size_q": self.hidden_size_q,
+            "hidden_size_kv": self.hidden_size_kv,
+            "hidden_size": self.hidden_size,
+            "num_attention_heads": self.num_attention_heads,
+            "attention_num_kv_heads": self.attention_num_kv_heads,
+            "word_window_size": self.word_window_size,
+            "key_query_norm": self.key_query_norm,
+            "key_query_norm_per_head": self.key_query_norm_per_head,
+        }
+
+
+@dataclass
+class DecoderHATModelConfig(TransformerHATModelConfig):
+    def __init__(
+        self,
+        num_attention_heads: int,
+        num_key_value_heads: int,
+        sliding_window: int,
+        cross_attention_config: CrossAttentionConfig,
+        cross_attn_every_layer: bool,
+        **kwargs,
+    ):
+        super().__init__(
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            sliding_window=sliding_window,
+            **kwargs,
+        )
+        self.cross_attn_every_layer = cross_attn_every_layer
+        self.cross_attention_config = cross_attention_config
+
+    def to_dict(self):
+        config_dict = super().to_dict()
+        config_dict["cross_attn_every_layer"] = self.cross_attn_every_layer
+        config_dict["cross_attention_config"] = self.cross_attention_config.to_dict()
+        return config_dict
+
+    @classmethod
+    def from_dict(cls, config_dict, **kwargs):
+        config_dict = config_dict.copy()  # Avoid modifying the original dict
+        config_dict.update(kwargs)  # Apply overrides
+        dict_config = config_dict.pop("cross_attention_config", {})
+        cross_attention_config = CrossAttentionConfig(**dict_config)
+        config_dict["cross_attention_config"] = cross_attention_config
+        return cls(**config_dict)
+
+
+@dataclass
+class EncoderHATModelConfig(TransformerHATModelConfig):
+    def __init__(
+        self,
+        cross_attention_config: CrossAttentionConfig,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.cross_attention_config = cross_attention_config
+
+    @classmethod
+    def from_dict(cls, config_dict, **kwargs):
+        config_dict = config_dict.copy()  # Avoid modifying the original dict
+        config_dict.update(kwargs)  # Apply overrides
+        dict_config = config_dict.pop("cross_attention_config", {})
+        cross_attention_config = CrossAttentionConfig(**dict_config)
+        config_dict["cross_attention_config"] = cross_attention_config
+
+        return cls(**config_dict)
+
+    def to_dict(self):
+        config_dict = super().to_dict()
+        if self.cross_attention_config:
+            config_dict["cross_attention_config"] = self.cross_attention_config.to_dict()
+        return config_dict
+
+
+@dataclass
+class HATArchitectureConfig(PretrainedConfig):
+    model_type: str = "hierarchical_autoregressive_transformer"
+
+    def __init__(
+        self,
+        special_token_dict: dict | None = None,
+        encoder_config: EncoderHATModelConfig | None = None,
+        backbone_config: TransformerHATModelConfig | None = None,
+        decoder_config: DecoderHATModelConfig | None = None,
+        model_type: str = "hierarchical_autoregressive_transformer",
+        eos_token_id: int = 192,
+        max_word_size: int = 100,
+        sliding_window: int = 768,
+        max_position_embeddings: int = 262144,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.encoder_config = encoder_config
+        self.backbone_config = backbone_config
+        self.decoder_config = decoder_config
+        self.model_type = model_type
+        self.eos_token_id = eos_token_id
+        self.max_word_size = max_word_size
+        self.special_token_dict = special_token_dict
+        self.transformers_version = "4.46.3"
+
+        # set these for out of the box vllm inference
+        self.architectures = ["HATDecoderForCausalLM"]
+        self.sliding_window = sliding_window
+        self.max_position_embeddings = max_position_embeddings
+        self.torch_dtype = "bfloat16"
+
+    @classmethod
+    def from_dict(cls, config_dict, **kwargs):
+        """
+        Instantiates a HATArchitectureConfig from a Python dictionary of parameters.
+
+        Overrides the base `from_dict` to correctly handle nested config objects.
+        """
+        config_dict = config_dict.copy()  # Avoid modifying the original dict
+        config_dict.update(kwargs)  # Apply overrides
+
+        # Pop and instantiate nested config dictionaries
+        encoder_dict = config_dict.pop("encoder_config", {})
+        backbone_dict = config_dict.pop("backbone_config", {})
+        decoder_dict = config_dict.pop("decoder_config", {})
+
+        # Instantiate nested configs
+        encoder_config = EncoderHATModelConfig.from_dict(encoder_dict) if encoder_dict else None
+        backbone_config = TransformerHATModelConfig.from_dict(backbone_dict) if backbone_dict else None
+        decoder_config = DecoderHATModelConfig.from_dict(decoder_dict) if decoder_dict else None
+        special_token_dict = config_dict.pop("special_token_dict", {"<|eot_id|>": 192})
+        max_word_size = config_dict.pop("max_word_size", 100)
+        return cls(
+            encoder_config=encoder_config,
+            backbone_config=backbone_config,
+            decoder_config=decoder_config,
+            special_token_dict=special_token_dict,
+            max_word_size=max_word_size,
+            **config_dict,
+        )
+
+    def to_dict(self):
+        config_dict = {}
+        if self.encoder_config:
+            config_dict["encoder_config"] = self.encoder_config.to_dict()
+        if self.backbone_config:
+            config_dict["backbone_config"] = self.backbone_config.to_dict()
+        if self.decoder_config:
+            config_dict["decoder_config"] = self.decoder_config.to_dict()
+        config_dict["model_type"] = self.model_type
+        config_dict["transformers_version"] = self.transformers_version
+        config_dict["auto_map"] = {"AutoConfig": "config.HATArchitectureConfig", "AutoModelForCausalLM": "model.HATForCausalLM"}
+        config_dict["special_token_dict"] = self.special_token_dict
+        
+        # print these out to the config for vllm
+        config_dict["max_word_size"] = self.max_word_size
+        config_dict["sliding_window"] = self.sliding_window
+        config_dict["max_position_embeddings"] = self.max_position_embeddings
+        config_dict["torch_dtype"] = self.torch_dtype
+        config_dict["architectures"] = self.architectures
+        return config_dict
+
+
+class EncoderHATModel(nn.Module):
+    def __init__(self, config: HATArchitectureConfig, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.config = config

config.yaml

@@ -0,0 +1,96 @@
+encoder_config:
+  vocab_size: 256
+  hidden_size: 2048
+  num_hidden_layers: 6
+  num_attention_heads: 16
+  num_key_value_heads: 16
+  rms_norm_eps: 1.0e-05
+  intermediate_size: 5632
+  max_position_embeddings: 98304
+  rope_scaling:
+    rope_type: default
+  rope_theta: 100000
+  mlp_bias: false
+  use_cache: true
+  sliding_window: 768
+  transformers_version: null
+  key_query_norm: false
+  key_query_norm_per_head: false
+  is_neox_style: false
+  cross_attention_config:
+    hidden_size_q: 8192
+    hidden_size_kv: 2048
+    hidden_size: 8192
+    num_attention_heads: 64
+    attention_num_kv_heads: 64
+    word_window_size: 1
+    key_query_norm: false
+    key_query_norm_per_head: false
+backbone_config:
+  vocab_size: 0
+  hidden_size: 8192
+  num_hidden_layers: 80
+  num_attention_heads: 64
+  num_key_value_heads: 8
+  rms_norm_eps: 1.0e-05
+  intermediate_size: 28672
+  max_position_embeddings: 12288
+  rope_scaling:
+    rope_type: llama3
+    factor: 8.0
+    original_max_position_embeddings: 8192
+    low_freq_factor: 1.0
+    high_freq_factor: 4.0
+  rope_theta: 500000
+  mlp_bias: false
+  use_cache: true
+  sliding_window: null
+  transformers_version: null
+  key_query_norm: false
+  key_query_norm_per_head: false
+  is_neox_style: true
+decoder_config:
+  vocab_size: 256
+  hidden_size: 2048
+  num_hidden_layers: 4
+  num_attention_heads: 16
+  num_key_value_heads: 16
+  rms_norm_eps: 1.0e-05
+  intermediate_size: 5632
+  max_position_embeddings: 98304
+  rope_scaling:
+    rope_type: default
+  rope_theta: 100000
+  mlp_bias: false
+  use_cache: true
+  sliding_window: 768
+  transformers_version: null
+  key_query_norm: false
+  key_query_norm_per_head: false
+  is_neox_style: true
+  cross_attn_every_layer: true
+  cross_attention_config:
+    hidden_size_q: 2048
+    hidden_size_kv: 8192
+    hidden_size: 2048
+    num_attention_heads: 16
+    attention_num_kv_heads: 16
+    word_window_size: 1
+    key_query_norm: false
+    key_query_norm_per_head: false
+model_type: hierarchical_autoregressive_transformer
+transformers_version: 4.46.3
+auto_map:
+  AutoConfig: config.HATArchitectureConfig
+  AutoModelForCausalLM: model.HATForCausalLM
+special_token_dict:
+  <|begin_of_text|>: 250
+  <|start_header_id|>: 251
+  <|end_header_id|>: 252
+  <|eot_id|>: 192
+max_word_size: 100
+sliding_window: 768
+max_position_embeddings: 98304
+torch_dtype: bfloat16
+architectures:
+- HATDecoderForCausalLM

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.46.3"
+}

model-00001-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a3564d64a71a648587cca3ca4409faf2ee040001b655e45ea7175202206504b
+size 4992458720

model-00002-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66578b63e7691f3a48e2256cc3ce37aa8be95c95f1f1cbc8fd9dd7773b2de95e
+size 4966123144

model-00003-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8af3cc2abf296cc6b5f6b07037052d8dfe01ff3e33d19350f8c355d09f07b8c2
+size 4362142896

model-00004-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:728ee67efaa02800d1d515992a251eb7bd60e0be8f4354a6026ce58e5ecb7dc0
+size 4966188912

model-00005-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad5b0b1350aa81b6b33e270c534583376d465e77dc99e6e1d530962b763139b9
+size 4362142896

model-00006-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2e92e4e91226c5b2df7359b490d5630e66f62c6774f99a668f9f793120c6630
+size 4362142896

model-00007-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49e85b28038dda70c6e28461665832df278eb68d61b086e4f0e66be740db4593
+size 4966188912

model-00008-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2f4ae5b8707a89aa2ec4ce69bc94f4c56b405a42c37b45a82dfe7b2703168fa
+size 4362142912

model-00009-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff1917704aae0063ec94e68aeab2e525cb5f581d36b956ede5ce871a289ebf44
+size 4362142904

model-00010-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:343c851aa5fd255bc9436947900060ea5738edb0259d04a6bcd16389828c4ad9
+size 4966188928

model-00011-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0f76ba1ffe4ff51f3634bf99bfcbb3bd5b65113742a931e5851fd13a1bad571
+size 4362142904

model-00012-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65eec13d9fcdd587bcc68b0fd26689630374378a609fe1aac2ce7a254754edbe
+size 4362142904

model-00013-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1eb3174f47d78c6e3fbd7aaac4f67bdc4e2a8ef93a729619721c1c6d67829cb2
+size 4966188928

model-00014-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf33ede7924c59c413cd5ca6a807e2d3bf785884617c1415a3010dea1e5c3bec
+size 4362142904

model-00015-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e754e308dbb6857fa72a9dcd5debec5668ec34e100e1959d4c31d34bc29c082
+size 4362142904

model-00016-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1dd2492d339b49223d5ff1453e21892133190c384d854187068ecfd27a4401f
+size 4966188928

model-00017-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e6baa0e48bf614903a8a9229fd8282042cd1715a109188405c899fa7b26e9b3
+size 4362142904

model-00018-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a77cc13fb9f95a201bcd57c7809c33b0651ff2509ab3dead7674cd7ca0a21e3c
+size 4362142904

model-00019-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:449f8e5949f4eb5b93c5972fc4782ca4703a0f42ee9fb78241289a7a6b0efc3d
+size 4966188928

model-00020-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee24f74535dfdde4df64897a65d854262cb484f083babc15a3f2229d34075179
+size 4362142904

model-00021-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:041a750c22a23c3ae802f4859b8b8f8d6a63fab078410ef7f514164893ba7671
+size 4362142904

model-00022-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8025479f96daea63dda480f9a2487febf8d3215e7b74fb403c35a6bf97a5f1d2
+size 4966188928

model-00023-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec93af46c20a36970a5042f62002441a271bb7ef051e07f7280640535c0e3ab6
+size 4362142904

model-00024-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f6964c6991c698480576cd213ee068fe713be7bc80397ccf7b8d5c6d5f0d467
+size 4362142904

model-00025-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39a8c532fe617953ed16b9ae77a5915290ed7e4e4c953f5d1b4a05bdaccf04bc
+size 4966188928

model-00026-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15a05861a07ea306675734de88667deae1e941275f4ebfa50c256253fd3bf052
+size 4362142904

model-00027-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef602e77a4af97a880e200f41093e86ee7fa9b670d811aac82ab2fa527cd2666
+size 4362142904

model-00028-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a126e15a8fbd4f52e1956c3394ae222767d1003c1d4c20b566d300cf58396960
+size 4966188928

model-00029-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8243868cf79176d8f28676763529ff32e0c44371488bbcc12d4e9dd90d2cf04d
+size 4362142904

model-00030-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa82277c1bfbfdef24585834a0cfe07883d7445dc1d31edbc48bbcb270dd16f1
+size 4362142904

model-00031-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54d0990bf736c86e819e98c4d9c1e9bb6bc5fc9f6b54f365a8f27fba649ff534
+size 4966188928

model-00032-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3bb9661ddf9defbfe47620a010a6efb77702597c502f4a93bda9e45b9195079
+size 4362142904

model-00033-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:354fd0af125ba4a1691634cb7b8eec00fe57efffc8e3dde2ce602bb14d9bad2a
+size 4362142904

model-00034-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3d177f40e8c999b1c2ca7c158183ef0276c6eea29c69730b01a52f757525b05
+size 4966188928

model-00035-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76d33fd4c8ca89e27c300ed88dadfcd360924b6f833de3969e6c2f87e9088c64
+size 4362142904

model-00036-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9d7d6f9fbbe0f57e61e872676c18bfc4f530e56b57dfb9d888e6bce2053d6d1
+size 4362142904

model-00037-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e77d304bff6efcaa039d48d962aa0e29cbf7025aeb3687a506f24636659cca39
+size 4966188928

model-00038-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e489cc420b93c994a877faf3fe9149d26484b6a1a028071b2a5caacaf7ef1350
+size 4362142904

model-00039-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6dcd371dd6a2b91b0264e9c95f6dad3d56520e498e6dae5fef9de54319b56c9
+size 4362142904

model-00040-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71d773ade2594bd860f813b1f9e9ee9ebf23c0799622cf442720da466cadc399
+size 4966188928

model-00041-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:692446421bd2399a65fe5a3de491a309d02123b7054b34ea48ca0d39208bde7e
+size 4362142904

model-00042-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acff5db3dbd2523fa33075fa6a832bf0524d59c535e4d2d94172321d5d2faae4
+size 4362142904

model-00043-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44687293209be69938c912920e387b079fbec59c14e26e46a37d2a0d26b379df
+size 4966188928

model-00044-of-00061.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bddb88ca0fab9fd7ef924f575a1c54a720619a668bc0885e07a212d8fd26942
+size 4362142904