The final part in this three-part series. If you have not checked out parts one and two, we suggest checking those out as we are going to continue where we left off.
If you are following on from the last blog, then you should have all the spaces, connections, and GitHub information already set up.
In this final part, we are going to play the role of the developer getting another change request. This time it's to include a new dataset from the GitHub data connection - repositories - and land them in Snowflake with the same conformations (uppercase column and table names).
Starting in the development project
We are going to start in our development project. Make sure you are back on the main branch of the project. If you are following along, you will be on the CR0001 branch - this is fine, but for completeness, head back to the main branch and apply changes. Prepare all these before beginning to make any additional changes to this new change request from the business.
Adding the repositories table
On the onboarding task, in the development pipeline, open up the 'select source data' and search all the datasets you have metadata for in the connector, and add the 'repositories' table.
Press Save. You should have a new table in the project. When you validate, you will see the new table - prepare this change on the onboarding task.
Updating the storage task
While this is preparing, go to the storage task. You will initially only see the two datasets - you will need to go in and out of the select source data for the UI to update the new tables in the onboarding task prior to this one.
Validating the project changes
Validate the project, check your new table was added, and prepare this change.
In Snowflake, within the DEV schema, we can see the changes. Note this time we didn't have a branch schema that we are working in - we are just using the 'main' branch of the project as an alternative method to developing changes.
Our production will NOT have these changes yet, so how can we publish these changes to production.
We could use the same method as the second part, but we are going to use the Qlik API to export and import the project. I am going to use Postman, but you can use an API tool of choice. You will also need an API key from the Qlik admin section to be able to make API calls. I will assume you have this already.
API deployment steps
The following steps will be required to promote this new table change from development to production using APIs:
Step 1: Get the development project ID
Get the project ID of the development project. This can be found in the URL when in the project.
Step 2: Make the export request
Make a POST request to the following endpoint api/v1/diprojects/{DEVprojectId}/actions/export
Step 3: Save the ZIP response
You will receive a ZIP binary, save this response to a file.
Step 4: Get the production project ID
Get the project ID of the production project, again, can be found in the project URL.
Step 5: Import to production
Make a POST request to the following endpoint api/v1/diprojects/{PRODProjectID}/actions/import with a form-data 'zip' where the file is attached.
You should get 200 OK with an empty JSON object returned.
Step 6: Validate in production
Head to the production project, check the onboarding task, you will see the new table REPOSITORIES there. If you validate, you will see it showing as a new dataset in the production pipeline.
Step 7: Prepare the production project
Prepare the project using the following endpoint POST api/v1/diprojects/{PRODProjectID}/actions/prepare.
You will receive an action ID as a response. You can poll this endpoint GET api/v1/diprojects/actions/{current_action_id} to check the status of the prepare.
Step 8: Run the production pipeline
Run the pipeline using the UI or via the POST /api/v1/dataprojects/{prod_project_id}/actions/start endpoint.
This will start the pipeline back up in production.
Deployment complete
That's it. You've made it! I hope you have learned something on how QTC CI/CD features can be used in your business to have development and production pipelines.
Get expert help with your QTC API implementation
Need help setting up CI/CD in your Qlik Talend Cloud environment? We've helped over 250 UK businesses get more value from their data. We specialise in Qlik, Snowflake and API automation.
Book a demo call today to see how we can help you implement proper CI/CD processes for your QTC pipelines.
Comments