CeLLaTe-tapt_ulmfit-LR_2e-05

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on the Mardiyyah/TAPT_data_V2_split dataset. It achieves the following results on the evaluation set:

• Loss: 1.0978
• Accuracy: 0.7612

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.426	1.0	21	1.2594	0.7324
1.405	2.0	42	1.2467	0.7366
1.3785	3.0	63	1.2092	0.7403
1.364	4.0	84	1.2270	0.7360
1.3309	5.0	105	1.2189	0.7399
1.3407	6.0	126	1.2297	0.7370
1.3141	7.0	147	1.1691	0.7421
1.3039	8.0	168	1.1879	0.7474
1.2869	9.0	189	1.1552	0.7452
1.2759	10.0	210	1.1660	0.7483
1.2406	11.0	231	1.1348	0.7519
1.2622	12.0	252	1.1563	0.7457
1.271	13.0	273	1.1812	0.7414
1.259	14.0	294	1.1268	0.7505
1.2611	15.0	315	1.1653	0.7402
1.2179	16.0	336	1.1489	0.7458
1.2283	17.0	357	1.1930	0.7389
1.2087	18.0	378	1.1431	0.7512
1.2244	19.0	399	1.1522	0.7454
1.216	20.0	420	1.1481	0.7475
1.1772	21.0	441	1.1044	0.7527
1.1823	22.0	462	1.1526	0.7462
1.1769	23.0	483	1.1515	0.7464
1.165	24.0	504	1.1232	0.7523
1.1894	25.0	525	1.1750	0.7442
1.1824	26.0	546	1.1338	0.7470
1.1846	27.0	567	1.1422	0.7526
1.1713	28.0	588	1.1659	0.7448
1.1499	29.0	609	1.1670	0.7402
1.1797	30.0	630	1.1475	0.7501
1.1402	31.0	651	1.1668	0.7494
1.1691	32.0	672	1.1417	0.7485
1.1405	33.0	693	1.1255	0.7516
1.1515	34.0	714	1.1319	0.7486
1.1523	35.0	735	1.1606	0.7421
1.1479	36.0	756	1.1598	0.7477
1.1586	37.0	777	1.1303	0.7514
1.1431	38.0	798	1.1498	0.7470
1.1249	39.0	819	1.1198	0.7507
1.1488	40.0	840	1.0946	0.7597
1.1192	41.0	861	1.1658	0.7436
1.1422	42.0	882	1.1911	0.7411
1.1417	43.0	903	1.1499	0.7456
1.13	44.0	924	1.1271	0.7513
1.1321	45.0	945	1.1536	0.7503
1.1297	46.0	966	1.1400	0.7464
1.1201	47.0	987	1.1694	0.7456
1.1116	48.0	1008	1.1379	0.7496
1.1438	49.0	1029	1.1962	0.7400
1.1286	50.0	1050	1.1648	0.7470
1.1178	51.0	1071	1.1946	0.7389
1.1045	52.0	1092	1.1552	0.7498
1.1239	53.0	1113	1.1641	0.7462
1.1091	54.0	1134	1.1907	0.7471
1.0978	55.0	1155	1.1708	0.7463
1.1087	56.0	1176	1.1300	0.7515
1.1212	57.0	1197	1.1515	0.7500
1.1249	58.0	1218	1.1530	0.7510
1.1021	59.0	1239	1.1405	0.7530
1.1024	60.0	1260	1.1327	0.7536
1.1015	61.0	1281	1.1644	0.7499
1.1103	62.0	1302	1.1186	0.7507
1.1259	63.0	1323	1.1596	0.7465
1.088	64.0	1344	1.1625	0.7454
1.0948	65.0	1365	1.1463	0.7467
1.1121	66.0	1386	1.2079	0.7424
1.0971	67.0	1407	1.1519	0.7487
1.0748	68.0	1428	1.1570	0.7433
1.1075	69.0	1449	1.1388	0.7519
1.0945	70.0	1470	1.1673	0.7484
1.0833	71.0	1491	1.1329	0.7516
1.0875	72.0	1512	1.1723	0.7418
1.0915	73.0	1533	1.1537	0.7478
1.0776	74.0	1554	1.1326	0.7550
1.0866	75.0	1575	1.1435	0.7490
1.0952	76.0	1596	1.1409	0.7436
1.0995	77.0	1617	1.1387	0.7516
1.0897	78.0	1638	1.1622	0.7446
1.0837	79.0	1659	1.1246	0.7522
1.1172	80.0	1680	1.1339	0.7490
1.0764	81.0	1701	1.1524	0.7537
1.0661	82.0	1722	1.1239	0.7547
1.1066	83.0	1743	1.1721	0.7495
1.0817	84.0	1764	1.1139	0.7548
1.0748	85.0	1785	1.1500	0.7459
1.0927	86.0	1806	1.1703	0.7445
1.1006	87.0	1827	1.1875	0.7432
1.0793	88.0	1848	1.1600	0.7454
1.0794	89.0	1869	1.1200	0.7554
1.0834	90.0	1890	1.1317	0.7464
1.091	91.0	1911	1.1384	0.7517
1.0903	92.0	1932	1.1452	0.7500
1.0838	93.0	1953	1.1264	0.7534
1.092	94.0	1974	1.1442	0.7471
1.0868	95.0	1995	1.1712	0.7412
1.0804	96.0	2016	1.1599	0.7472
1.1127	97.0	2037	1.1481	0.7516
1.0712	98.0	2058	1.1194	0.7519
1.0723	99.0	2079	1.1521	0.7447
1.099	100.0	2100	1.1281	0.7482

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0