Abstract: In recent years, deep learning (LeCun, Bengio, & Hinton, 2015) has drawn interest in many fields. As optimism for deep learning grows, a better understanding of the efficacy of deep learning is imperative, especially in analyzing and making sense of educational data. This study addresses this issue by establishing a benchmark for a common prediction task – student proficiency in diagnosing patient diseases in a system called BioWorld (Lajoie, 2009). To do so, we compared deep learning to existing solutions, including traditional machine learning algorithms that are commonly used in educational data mining. The dataset consists of log interaction data collected from 30 medical students solving 3 different cases. A 10-fold cross-validation method was used to evaluate the predictive accuracy of each model. Interestingly, our results indicate that deep learning does not outperform traditional machine learning algorithms in predicting diagnosis correctness. We discuss the implications in terms of understanding the proper conditions for its use in educational research.