Mastering CI/CD in QTC: a three-part guide to deployment. Part 1: initial dev to prod

This article is the first in a three-part series covering CI/CD implementation in Qlik Talend Cloud. Following our introduction to GitHub CI/CD integration in QTC, we now present a comprehensive guide for establishing proper deployment processes.

The complete series explores:

1. An initial dev to prod migration
2. Using Space & GitHub CI/CD to promote changes
3. Using Export/Import API to promote changes

This guide demonstrates a recommended approach to setting up and using QTC with CI/CD methodologies, focusing on the initial migration from development to production.

My setup environment

I have the following setup:

• In Qlik, I have a dev & prod space with connections to a source (Github data) and target (Snowflake) with connection in their respective spaces and credentials.
• In Snowflake, I have 2 databases, OMETIS_LANDING and DEV_OMETIS_LANDING where I will be landing the data from my source into target.
  
  

Creating a project in development

To begin with, I am going to create a new project in the DEV space on QTC and I'm going to call it 'Github Ingestion (Via Pipeline)' as I'm using a pipeline project instead of a replication project (see my blog about cross pipeline architecture to why I've chosen to do it this way).

Now I have my blank Pipeline Project, I'm going to onboard some data from github api using the qlik connection.



And I'm just going to pick 2 tables, teams and users from the api endpoint. I'm going to just onboard the data and not worry about type 2 for the moment.



My project now looks like this:

Configuring the pipeline tasks

Landing task configuration

Within the 'Landing' task I am going to change the following within the settings & name of the task:

1. Name = Onboard Github Data
2. DATABASE = DEV_OMETIS_LANDING
3. TASK SCHEMA = GITHUB
4. PREFIX FOR TABLES= QTC_

And I'm going to prepare.

Storage task configuration

While that is preparing I'm going to change the Storage task:

1. Name = Stored Github Data
2. DATABASE = DEV_OMETIS_LANDING
3. TASK SCHEMA = GITHUB
4. INTERAL SCHEMA = GITHUB
5. PREFIX FOR TABLES= blank

And then prepare that.

If I look in snowflake, I can see all the assets QTC has created for those 2 tables. The 2x QTC_ prefixed

Onboarding tables, and the teams_ct and teams_current tables with supported views.

Promoting to production

Great, so we have got a development project setup. We can run this and develop any changes as required. The next step is to promote your pipeline to production. Here's how:

There are 2 simple steps:

1. Export project
2. Import project & update the 'bindings'

Exporting your project

You can export a project in a few places, I will use the three dots from the pipelines menu and press export.

You will received a Json file. This contains all the information about the QTC project. We can use this to import a new project. And to do this, you need to do the following steps:

Importing to production environment

1. Press Create New, then Import Project

2. Within the menu, you will need to populate the following:

• a. Select the exported JSON from the export steps.
• b. Change the name to remove the 'Imported_' bit
• c. Put it in the production space
• d. Change the Data platform from the DEV snowflake connection to the PROD snowflake connection
• e. Change the Source Connection from the DEV Source connection to the PROD source connection
• f. Leave Prefix for all schemas blank
• g. Within the database name overrides, remove the DEV_ bit in my case from DEV_OMETIS_LANDING To just
• OMETIS_LANDING (Our prod space)
• h. Change the warehouse if required.
• i. Press Import

3. You will have a new project, that looks exactly the same as your dev project, but with PROD connections.

4. Prepare the pipeline.

Verifying successful deployment

Now we can check in Snowflake the assets that have been created.

They will match the OMETIS_LANDING_DEV.GITHUB schema from the dev branch.

Implementing CI/CD in your organisation

This was one simple method you can use and incorporate into your business to governed QTC pipelines.

Stay tuned for the next two blogs on how we can promote changes using GitHub Integration and promote changes using the Qlik API.

Get expert help with your  QTC CI/CD implementation

Need help setting up CI/CD in your Qlik Talend Cloud environment? We've helped over 250 UK businesses get more value from their data. We specialise in Qlik, Snowflake and GitHub integration.

Book a demo call today to see how we can help you implement proper CI/CD processes for your QTC pipelines.