Developing a machine learning model to forecast migraine attacks

Thank you so much for the opportunity to talk about this project. At AAN, I presented a study that used a machine learning approach to forecast migraine attacks based on detailed clinical symptoms and the rapid number naming test. The reason why we were doing this study, or why we designed this study, was because for a lot of patients with migraines, they might frequently wonder, am I going to have a migraine attack soon? Or if they’re already in a migraine attack, whether this migraine attack would continue? Currently, there is no accurate way to forecast whether someone will have a migraine attack, or how long a migraine attack will continue. We did look into previously published studies, and there were several studies that had included machine learning approaches to forecast migraine attacks, but those studies utilized symptoms or features collected once daily to predict migraine attack over the next day, with room for further optimizing of the model performance. So, the design of our study was to utilize detailed clinical symptoms in an objective measure, in this case a rapid number naming test called the King-Devick Test (KDT). We did the test three times a day, and then designed a machine learning model that can forecast migraine attacks over the next six hours, rather than the next day, to build this accurate machine learning model that can forecast migraine attacks. That is kind of the background and the goal of this study.

Previously, we presented several studies, kind of our pilot study, to use this rapid number naming test (the King-Devick scores), and we found that there was a difference between the migraine attack phase and the interictal baseline; that the KDT score was slower during the migraine attack phase. So subsequently, we designed another study where we enrolled 30 participants and asked each individual to take a King-Devick score and also record very detailed symptoms that they are experiencing at the moment and which migraine state that they felt that they were in at that time. We did that recording three times a day over a four-month period. From 30 participants each recording daily three times a day for four months, we collected close to 5000 data points. So that’s why with a sufficient amount of data we have constructed this machine learning model to forecast migraine attack over the next 6 to 12 hours.

For our machine learning model approach, we used a leave-one-out approach. We used the dataset from 29 subjects for model training and then one subject for testing. This process was repeated 30 times. Our results showed that we were able to build several quite robust machine learning models, so the overall model performance had an Area Under the Curve (AUC) of 0.66 close to 0.7, which is the overall kind of performance of all of these 30 models that we constructed. However, when we look at each individual model, our most robust models had an AUC in the test set of 0.95 and accuracy of over 90%. Our top four models had accuracy over 90%. So that means that for the models that we built, the top performing models had robust performance and high accuracy. So, that was one important conclusion of our study. Another important conclusion of our study is that we did a variable importance analysis. We analyzed which features or variables were the most important for the model to make a prediction of whether someone will have a migraine attack over the next six hours or not. The most important variables were the current migraine state (the self-report migraine state that participants felt that they were currently at), time from last migraine attack, several different measures of the KDT scores, and also the current health state and sleep state were very important. In addition, several symptoms that participants experienced at particular states. So, for example, experiencing photophobia and phonophobia when they are having a premonitory or prodrome phase, or experiencing photophobia and fatigue during the migraine attack phase. Those symptoms were the most important for the model to predict whether someone will have a migraine attack over the next six hours or not. So, I think those were the most important take home messages. Using detailed headache characteristics or symptoms someone is experiencing at the moment, plus an objective measure that was recorded multiple times a day, we could develop a machine learning model that can accurately forecast whether someone will have a migraine attack over the next 6 to 12 hours using a robust machine learning approach.

We are planning to prospectively validate those most accurate models in a prospective study to see how good this model performs. I also wanted to point out that this is the benefit of using AI or machine learning models to forecast a migraine attack. This is because when we analyze our results from our previous studies or pilot studies, if we use a simplified approach (for example based on the KDT score alone), because there’s a lot of fluctuation in the KDT score even during a interictal baseline phase when someone doesn’t have a migraine attack, there will be fluctuation of the KDT scores that we observe for some subjects, and the KDT score will continue to go up with time or go down with time. So, a simplified approach, like using a KDT score alone even during a premonitory phase or prodrome phase, might not be accurate enough to forecast migraine attack. So that’s why we decided to take this machine learning approach using multiple information, using baseline headache characteristics, demographic data, several questionnaires participants filled out a baseline, this multiple data point (including self-reported symptoms) and an objective measure, to construct this machine learning model that could yield the most accurate results. So, I think the study also speaks to the importance of incorporating multiple aspects of data and using the robust machine learning models to forecast a migraine attack. This is rather than taking a simplified approach to see whether someone will experience a migraine attack.