Student support and retention is an essential part of student success within higher education. The better the support universities are able to provide, the more likely the students are to continue with their studies and attain academic success. In turn, this supports the university's performance metrics, such as the Teaching Excellence Framework and funding.
Since the 2010-11 academic year, non-continuation rates have been slowly rising, with almost 10% of students not continuing with their academic studies after the first year as of 2016-17. This shows that universities need to do more to support their students in this changing environment and use the data they’ve likely already collected to drive real impact. This matters for universities because the more they can support students, the better the retention rates and student success. This means that universities will be able to retain the fees those students are paying as well as improve the student’s education and future careers.
Royal Holloway University partnered with Ometis to use Qlik Predict to identify year 1 undergraduate students who were at risk of not progressing to year 2 in order to deliver more personalised support. Using Qlik Predict, the University developed a machine-learning model that analyses attendance, Virtual Learning Environment (Moodle) activity, academic performance, and demographic data to predict progression at the beginning of Term 2 in a student's first year — a point at which Term 1 assessments have been completed, and there is still sufficient time to reach out to students and provide meaningful support.
Four years of historical data was cleansed, transformed, and enhanced through feature engineering to create indicators such as Virtual Learning Environment (Moodle), unauthorised absence rates, and average module scores. The final model achieved 89% accuracy against the test data, with SHAP analysis revealing the most influential factor to be Assessment Attempted, highlighting the importance of online engagement.
Once validated, the model was applied to data captured at the end of Term 1 assessments in the current academic year to identify those least likely to progress and the specific factors affecting them. At the end of the academic year, we compared this against actual results to obtain 86% accuracy against the 2024/25 data.
If this were in production, once those at-risk students had been identified, this would allow support teams to intervene earlier and tailor actions to each student’s needs - for example, addressing low VLE engagement, or repeated absences.
By turning existing data into actionable insights, Royal Holloway can provide more targeted, effective support and improve student retention, satisfaction, and engagement. The approach is scalable and could be adopted by universities worldwide facing similar challenges.
Royal Holloway University wants to be better able to support their first year students and identify any factors that may be negatively impacting the student’s likelihood of progressing into their second year. Identifying these impacting factors earlier on is important so that they can intervene earlier and provide personalized support for the student. They are currently only able to provide more generalized support, such as encouraging students to attend lectures or complete online quizzes, when for many students there are other factors at play where they need more specific support. Predicting the student’s progression allows the university to identify which students are at risk of not progressing and what those key factors are which are impacting them.
This is the perfect scenario for using Qlik Predict as this product uses machine learning to find patterns in the data and predict outcomes based on future data.
First, we had to decide exactly what question we’re trying to answer as this is key to generating a good predictive model. To do this we must determine the following:
What is the event trigger? This is the event or action which prompts the creation of a new row of data. In our scenario, this is someone being a first year student at Royal Holloway University.
What is the target we are trying to predict? This is the outcome that we want to predict which should also include a timeframe for the outcome. We want to predict the outcome of a first year student progressing into their second year based on the first semester of study.
What features do we want to include? Features are the columns in your data which are used to predict the target. These should only include data that you’ll know at the point of prediction – for example, when training the model we wouldn’t include any data beyond 10th of January as this is after the prediction point and this would create data leakage. We will include many features such as demographics, attendance records, VLE logins, VLE assessments attempted % and course information.
What is our prediction point? This is the point in time where we stop collecting data and make the prediction for the target. We have chosen this to be the 10th of January, so this is when we would stop collecting data about a student.
In answering the above questions, we can now formulate the exact question for our student progression scenario:
“As of 10th of January in the student’s first year, will they progress into their second year?”
This is a binary classification problem as there are only two possible outcomes – yes or no.
Data cleansing
Data preparation: this is the first, and arguably most important, step in the machine learning process to ensure we have a robust training dataset.
A good understanding of the data is essential so that we can work out what steps need to be taken to cleanse the data, which may include removing or replacing nulls, filtering down the dataset, aggregation and feature engineering.
We had four years of historical data available to us on which to train the model.
First, we excluded any students that weren't first year, weren't Mainstream intake or were not commencing their studies in September with the main cohort. This is because the data is either not relevant or would potentially skew the predictive model.
We then filtered the data so that we only include data up until the 10th of January, our prediction point. This prevents data leakage.
Then we removed a number of fields where they are considered unreliable features. These fields are removed for the following reasons:
The next step is data transformation, including replacing null values, grouping values and feature engineering.
Replacing null values can be done in a variety of ways depending on the data in the field. For numeric fields, it is best to replace nulls with the average value, whereas for categorical fields, it is better to either use the most frequent value or replace it with ‘unknown’ or ‘other’.
Grouping values can help simplify the model where a field has a variety of low level values which can be grouped into a higher level category. In our transformation, we have grouped [Extension Reason] into broader group categories, where several reasons such as ‘Illness’, ‘Injury’ and ‘Medical Condition’ could all be categorized as ‘Health’ as a reason for requesting an extension on an assignment.
Next, we focused on feature engineering. This is the process of creating new feature fields from existing ones, which allows you to gain additional information to improve the predictive model. Qlik Predict will auto-engineer features from date and time fields, parsing the individual components into separate fields including day, week, month, year, etc. Beyond this, it is possible to engineer further date features manually in the data transformation process if you want to add semester, season or financial year.
Using Attendance data we were able to create the following features:
From the VLE data we created the following:
From Mark Management System data we were able to create the following:
And from assessment extension data (where students have requested an extension on an assignment), we created:
Note: It is important to have open conversations with business experts to understand and validate the data. More often than not, they are able to make assumptions about which features will impact the target outcome. For example, if a student does not make any submissions on Moodle, then they will not pass their first year and be able to progress into year 2. This indicates that Moodle submission rates are a key indicator of if a student will progress or not and engineering further features from this data may provide additional insights. There may also be university specific logic which wouldn’t apply in other scenarios, such as the unique way extensions work in the first year at Royal Holloway.
Before we put our training data into a Qlik ML Experiment, we split our data into two outputs – training data and testing data. This can be done in a variety of ways and proportions. In this case, we are using 75% of the data to train with and the remaining 25% as test data, which get used to validate the model. Whilst this is not an essential step as the ML Experiment will do this as part of training the model using a 80-20 split, it can be useful to create this split manually if you want to test using different proportions, or always want to use the same set of data to test with when making adjustments to the data prep and transformation steps. Otherwise, the 20% used to test will differ each time you run the experiment.
Now that our training data has been transformed, we can use it within an ML Experiment. When running the experiment for the first time, we include all features in the dataset and select [Progress Flag] as our target field which we want to predict. We also have to go through all the fields in the model and make sure they are being recognized as the correct data and feature type (e.g. string/float/numerical/categorical/etc.) and fix these if required.
Once the experiment has run, we are shown a number of different models which Qlik has rated with an F1 score.
The highlighted model is the best model according to Qlik as it balances accuracy and prediction speed.
Selecting this model allows us to view which features have been included as impacting the prediction. Qlik states which features have been dropped and why. Many features have been dropped due to ‘Low permutation importance’ which means they do not have a significant impact on the model performance so they can be dropped without negatively affecting the prediction accuracy. Others, such as StudentID, have been dropped due to high cardinality.
We are also able to see if any features have been automatically engineered by Qlik. For example, Degree Aim and Department have both been encoded as they are categorical variables and these need to be converted to numeric fields to be included in the model. Different types of encoding are automatically chosen depending on how many unique values are in the field.
In the Models tab, we are also able to compare the performance of the suggested models. Sometimes we wouldn’t want to just pick the best model as suggested by Qlik, and may instead want to pick the most accurate model if we are less bothered about how fast the model runs. In this scenario, the models are all similar in accuracy and speed, so we can deploy the recommended best model.
Using our chosen deployed model, the ‘Test’ data can now be used to create a prediction. We output the SHAP and Apply datasets alongside our prediction output as the SHAP calculations in these datasets can be used to understand which features are the key indicators of non-progression for each student.
After creating the prediction output files, these are loaded along with our ‘Apply’ dataset into a Qlik Cloud app so that we can analyze the output of the prediction and test the accuracy of the model, beyond the automatic accuracy testing done in the ML experiment.
We can also analyse the overall % predicted to progress and not progress by factors such as Department, which can show if there are positive/negative trends amongst certain departments.
Since the Progress Flag field was left in our ‘Apply’ dataset, we have the actual outcome available to compare the prediction against. Loading in the ‘Prediction’ data gives us the predicted outcome, plus the probability of each outcome, all aggregated by StudentID so that we can join it to our ‘Apply’ data.
In the front end of our app, we can now use the actual Progress Flag value and the Progress Flag Predicted in a measure to check the accuracy.
Overall, we had an outcome of 89.9% correct predictions against our holdout data. This means that our features were good indicators of the progression outcome of a student.
The SHAP output can be used to see the impact of each feature on the Progression Flag and even determine the top x impacting factors by student. Counting the number of students by the top impacting factor gives a good overview of the key features which are impacting a student’s progression.
We discovered that the top factor impacting the largest number of students was Assessments Attempted%. From this we could assume that the fewer attempts the student made on a Moodle test/assignment would indicate a lower likelihood of the student progressing, but could dive further into details to determine if this assumption was correct.
Using SHAP scores we created the below chart, with the x-axis showing the % of Moodle content attempted, and the y-axis showing the likelihood of a student not progressing. This tells us that those students that attempt between 0-10% of Moodle content have a likelihood of not progressing of over 90%.
Now that we were confident in the predictions of our model, we used the latest 2024/25 academic year data to run the model on, giving our first batch of actionable insights. This data wasn’t used in the training of the model and we did not know the outcome of the students’ progression when we ran the predictions.
In Qlik, we chose our model deployment and created a new prediction from the latest dataset. Again, we include the Apply and SHAP data in the output and then load these into a Qlik app. From this, we can take a similar approach to when testing the model and create a number of charts showing the top impacting features by student and their likelihood of progressing.
Of the 3169 students included in the 2024/25 data, we predicted the number of individuals who would progress and those who would not. Comparing with the actuals, we got the following:
The true values are the ones where the predicted outcome was the same as the actual outcome. This gives us an overall accuracy of 86%. The true values are the ones where the predicted outcome was the same as the actual outcome. This gives us an overall accuracy of 86%. It is worth noting that despite this relatively high accuracy rate, the model can be improved further - the accuracy of predicting progressors was 90%, whilst the accuracy of predicting non-progressors was 64%. This disparity prompted us to look further into the reasons behind it, and led us to conclude that further work on the model to split it by subject would be beneficial.
Of the false positives, we know that only a small number left as a result of academic failure..., so others may have either retaken year 1 or left as a result of a factor which could not be predicted such as health/family issues. Of the false negatives, a proportion of these progressed after exam resits.
Royal Holloway University can change the way they intervene and provide support in a more personalised way to each student, as they can see the key factors which are indicative of the student struggling. For example, if the student’s top factor is Moodle Attempted %, the university are then able to look at the value for this factor – if the value is low or zero, this indicates that the student may not be attempting to complete all the Moodle tasks, or they may be struggling to access Moodle in the first place. The university can then tailor their approach to this student and support them in accessing Moodle and the content. If a student’s top factor is Unauthorised Absence % then support staff can reach out and find out what’s impacting this – are they suffering with health issues? Are they unable to travel to the university? Or is there another reason they are absent without authorisation?
The proposed change could positively impact students at Royal Holloway University by improving the institution’s ability to deliver personalised support at critical points in the academic journey, thereby supporting student wellbeing, retention, and continuation into the following academic year.
The data is already available and with Qlik Predict can be turned into actionable insights, building upon existing support provision to enable earlier, more targeted support that promotes student wellbeing and success. First-year student progression into second year is key to driving both student and university outcomes.
If this had been productionised at the time of prediction, Royal Holloway could have reached out earlier to the 509 students predicted not to progress, offering more personalised and targeted support at the point they needed it most. Whilst 182 of them progressed anyway, if the intervention helped even 50% of the remaining 327 students who were predicted to and did actually not progress, then 164 additional students could have continued their studies. As a secondary benefit, with each retained student representing £9,250 in annual fee income, this would also have totalled over £1.5m for all 164 students.
This is not a unique problem to Royal Holloway University and could be applied to universities across the country and world. Most institutions are already collecting this data about attendance, online portal use and course information, so the opportunity is there to use this to drive real actionable change using Qlik Predict.
Royal Holloway is one of 25 universities we work with, so we know the systems your student data lives in and, more importantly, what it takes to act on it before students slip away.
If you're wondering whether this could work with your data, that's usually the best place to start a conversation. Let's have a chat with us and we'll talk it through.