Mastering CI/CD in QTC: a three-part guide to deployment. Part 2: using GitHub version control

Welcome back to our guide on mastering CI/CD in Qlik Talend Cloud. This is part two of our three-part series where we'll show you how to use GitHub version control to promote changes from development to production.

If you haven't read part one yet, we recommend checking it out first as we'll be building on what we covered there.

In this article, you'll learn how to:

1. Connect your Qlik Talend Cloud projects to GitHub for version control
   
2. Create development branches for implementing changes
   
3. Test changes in isolated schemas
   
4. Merge and promote those changes to production
   
5. Apply remote changes to your production environment

Prerequisites for Qlik Talend Cloud CI/CD workflow

Before we begin, make sure you have the following:

• In Qlik: dev and prod spaces with connections to a source (GitHub Data) and target (Snowflake) with connection credentials in their respective spaces
• In Snowflake: two databases, OMETIS_LANDING and DEV_OMETIS_LANDING for landing data from source to target
• In GitHub: a new repo with just a readme file
• A Qlik developer access token with permissions to connect to QTC

If you need help setting up GitHub version control in Qlik Talend Cloud, check out our blog post on "How to set up GitHub version control in Qlik Talend Cloud" and follow steps 1 and 2.

At the end of part one, you should have two projects both called "GitHub Ingestion (Via Pipelines)" in two data spaces: dev and prod. Having the same name for both projects is crucial for this workflow to function correctly.

Connecting QTC dev project to GitHub version control

Let's start by going into the dev space and opening the dev project.

Next, we'll connect this project to version control:

1. Click 'Connect To Version Control' for the dev project
2. Select the repo you created earlier
3. Leave the directory path as default (or change if required)
4. Add an initial commit message such as 'Initial Commit'
5. Press OK

You'll notice the project now shows 'Main' next to it, and your GitHub repo has some new files.

Creating branches in Qlik Talend Cloud for isolated development 

Let's say a change request has come in asking for all landed table names in Snowflake to be uppercase as part of a data conforming process. We need to adjust our pipeline to accommodate this change.

We'll build this in dev first, then migrate to prod. Here's how:

Using the Qlik GUI, create a new branch:

• Branch Name: CR0001
• Branch Prefix for all schemas: CR0001_

After pressing OK, you'll see 'main' change to 'CR0001' and a new branch appears in GitHub.

Making changes to QTC pipelines in your development branch

Now we can make the required changes to our pipeline:

• One to uppercase table names
• One to uppercase column names

See all the assets QTC has created for those 2 tables. The 2x QTC_ prefixed

Onboarding tables, and the teams_ct and teams_current tables with supported views.

Testing changes in an isolated Snowflake schema

Before promoting to production, we can prepare and observe our target (Snowflake):

Notice a new schema was created where you can view all your new changes within that development branch.

In this case, the table names are now uppercase.

Using GitHub pull requests to promote QTC changes

Once you're happy with these changes and want to promote them to the prod project, follow these steps:

1. Commit the changes to the CR0001 branch via the Qlik UI:
   1. Add a meaningful commit message
   2. Commit and push

2. In GitHub, create a pull request (this feature isn't currently available within the Qlik UI)

3. You can build an internal process around this merge request, such as code reviews or a sign-off process

• For this demonstration, we'll push and merge the changes to main

Applying remote changes to Qlik Talend Cloud production 

Now that our changes are in the main branch, we need to update the production environment:

1. Go back to QTC and head to the prod project
2. Connect this project to GitHub using the same repo
   1. Since we used the same project name, it will sync with the repo status
   2. Important: Do NOT commit to main from the prod project
3. Within the GitHub menu, click 'Apply Remote changes'
   1. A menu will pop up showing all differences between prod and the main branch
   2. You can see all changes on the left with details in the text areas
   3. Select the checkbox to apply all changes, then click 'Apply remote changes'
   4. You should get a success message

Validating changes in production environment 

4. Before preparing the project, check the onboard tasks to confirm it correctly applied the state of the dev project to prod

As expected, it looks the same and is using the production source and target.

5. Validate to check the changes, then recreate/prepare those changes into prod

Confirming Snowflake changes in production 

6. Within Snowflake, you can now observe that the assets in the production schema have changed to uppercase.

CI/CD workflow summary for Qlik Talend Cloud

In this guide, you've seen how to promote changes from a dev pipeline to production using GitHub version control. While there are some initial setup steps to connect QTC projects with GitHub, the overall process is straightforward:

1. Create a branch in the dev pipeline, make and commit changes
2. Complete a pull request and merge changes in GitHub
3. Pull the changes from the master branch in prod, then prepare and run the project

In the final part of this series, we'll show you how to accomplish all of the above using only QTC APIs without the GitHub integration.

Get expert help with your  QTC CI/CD implementation

Need help setting up CI/CD in your Qlik Talend Cloud environment? We've helped over 250 UK businesses get more value from their data. We specialise in Qlik, Snowflake and GitHub integration.

Book a demo call today to see how we can help you implement proper CI/CD processes for your QTC pipelines.