Identifying potential drug-drug interactions (DDIs) before clinical use is essential for patient safety yet remains a significant challenge in drug development. We presented DDI-GPT, a deep learning framework that predicts DDIs by combining knowledge graphs (KGs) and pre-trained large language models (LLMs), enabling early detection of potential drug interactions. We demonstrated that DDI-GPT outperforms current state-of-the-art methods by capturing contextual dependencies between biomedical entities to infer potential DDIs. Through feature attribution methods, we show that our explainable deep learning (DL) models enhance the quality of explanations on the pathways and interactome networks. Using TwoSIDES as our benchmark dataset, DDI-GPT achieved the best performance of 0.964 in AUROC compared with existing DL methods. We also applied DDI-GPT to perform zero-shot prediction on 9,480 DDI records, encompassing 442 distinct drugs from the FDA Adverse Event Reporting System. DDI-GPT can attain a high accuracy of in 0.84 AUROC, with an improvement of 14% compared to the best previously published method. We explored model interpretations on predicted DDIs involving Bruton's tyrosine kinase (BTK) inhibitors and uncovered CYP3A-enriched signals underlying the contaminant use of BTK inhibitors with other drugs leading to toxicity. Altogether, DDI-GPT, implemented as both a web server platform and a software package, identifies DDI events and offers a deep learning tool for drug safety use with expandable features.