Javed Usman, MuhammadMuhammadJaved UsmanShafqat Ali, MuhammadMuhammadShafqat AliIqbal, AsmaAsmaIqbalDr. AZHAR MuhammadAslam, Shafqat MariaShafqat MariaAslamShabbir, IfraIfraShabbir2025-02-182025-02-182024Usman Javed, M., Ali Shafqat, M., Iqbal, A., Azhar, M., Aslam, S. M., & Shabbir, I. (10 Dec 2024). Transforming heart disease detection with BERT: Novel architectures and fine-tuning techniques. FIT 2024, Islamabad Pakistan.979833151050397983315105102473-75692334-3141http://hdl.handle.net/20.500.11861/10711The prediction of heart disease is crucial for effective prevention and treatment. However, extracting clinical infor-mation such as CAD, smoking, hypertension, hyperlipidemia, obesity, and family history of CAD from unstructured electronic health records (EHRs) poses significant challenges to clinicians. This research introduces a novel approach that leverages an ensemble of transfer learning algorithms combined with a multi-head attention mechanism to automatically extract heart disease risk factors from EHRs. Various deep learning models, including BERT, BioBERT, BioClinical BERT, RoBERTa, and XLNet, were initially trained on medical data sets and subsequently fine-tuned on the i2b2 clinical data set. Individual models delivered strong results, with RoBERTa achieving the highest accuracy of 95. 27% and an F1 score of 94. 94%. BioBERT, BioClinical BERT, XLNet, and BERT also performed well, with precision ranging from 94. 73% to 95. 03%. However, the proposed ensemble model with multi-head attention outperformed all, achieving an accuracy of 96.35% and the F1-score of 95.76%. These findings highlight the superior ability of the ensemble model to capture complex inter-dependencies between heart disease risk factors, making it a robust tool for clinical prediction.enCardiovascular DiseaseDetection Of Heart DiseaseFine-tuning TechniqueObesityClinical DataCoronary Artery DiseaseElectronic Health RecordsHyperlipidemiaF1 ScoreDeep Learning ModelsIndividual ModelsAttention MechanismTransfer LearningEnsemble ModelRisk Factor For Heart DiseaseMedical DatasetsMulti-head Attention MechanismSmoking StatusSupport Vector MachineHypercholesterolemiaNatural Language Processing ModelsNamed Entity RecognitionMasked Language ModelPrevention Of Heart DiseaseAttention HeadsBERT ModelText ClassificationClinical TextLanguage ModelWord EmbeddingConference Paper