
You use Snowflake and write SQL. Have you ever dreamed of a tool to perform integration tests on your SQL code? Dream no more, Karate Connect is here.
What is Karate Connect ?
You are probably familiar with the excellent open-source tool Karate, which is used for HTTP API test-automation. You write a feature (a simple text file using Gherkin Syntax and Karate DSL) and can execute it as an integration test using a simple command java -jar karate.jar myfirst.feature
🤩 !
Feature: my first feature Scenario: my first scenario Given url http://mybaseurl.com And path myresource When method get Then status 200 And match response == "Hello my resource"
Karate Connect is a fat JAR, also open-source, that includes Karate Core along with several extensions 🧩 to connect to other systems such as Snowflake, RabbitMQ, Kubernetes and DBT (extensions for Kafka and JDBC are coming soon 🤞).
* def result1 = <extension>.<value> * def result2 = <extension>.<feature>.<function>(args)
* json cliConfig = snowflake.cliConfigFromEnv * def rabbitmqClient = rabbitmq.topology.createClient({ host: "localhost", port: 5672 })
Karate Connect can also be conveniently used via Docker images 🐳.

Why Karate Connect ?
Lectra is a company that produces software and connected equipments such as cutters. All these devices send significant data to the cloud to offer services that include dashboarding and real-time applications.

A data pipeline has been developed for this purpose 🧑🔧 ! However, it is not straightforward as it appears. The initial pipeline was based on ETL architecture. For real-time needs, Apache Kafka was chosen. Kapoeira, another open-source tool developed by Lectra, is used to test Kafka Streams. More information and presentations can be found here, including a talk at Devoxx France 2024 in French.
Data is ultimately stored into a Snowflake database for dashboarding applications, thanks to Snowflake Snowpipe Streaming Kafka Connectors.
Recently, the ELT architecture has also been adopted for some batch processes to take advantage of the significant scalability of Snowflake’s computing power. The batches are written in SQL (as a Snowflake task) or in DBT.
But how to perform integration tests in this ELT architecture? DBT only offers unit tests.
Karate appears to be the good choice :
-
Snowflake already proposes HTTP API for SQL interactions,
-
Karate includes a powerful assertion engine and generates great HTML reports
-
Karate can call external tool using the syntax
karate.exec("<command>")
-
Karate DSL extensions can be developed for added simplicity
Thus, Karate Connect was born 🎊 !
How to run Karate Connect ?
Java usage
java -Dextensions=<ext1>,<ext2>... -jar karate-connect-<version>-standalone.jar <karate_args>
Docker usage
docker run --rm \
-v <features_path>:/features \
-v <reports_path>:/target/karate-reports \
-e KARATE_EXTENSIONS=<ext1>,<ext2>... \
lectratech/karate-connect:<version> <karate_args>
Snowflake examples (karate-connect
version 0.2.0
)
Configuration
{ "account": "xxx.west-europe.azure", "user": "MY_USER", "privateKeyPath": "/my-path/private-key.pem", "privateKeyPassphrase": "my-passphrase" }
{ "database": "MY_DB", "schema": "MY_SCHEMA" "role": "MY_ROLE", "warehouse": "MY_WH" }
Simple select
Feature: SELECT Background: * json cliConfig = read('classpath:cli-config.json') * json snowflakeConfig = read('classpath:snowflake-config.json') * string jwt = snowflake.cli.generateJwt(cliConfig) * json restConfig = ({jwt, cliConfig, snowflakeConfig}) Scenario: Select 1 cutter Given text statement = """ SELECT SERIAL_NUMBER, CUTTER_TYPE FROM CUTTER WHERE SERIAL_NUMBER='MY_VECTOR' """ And def response = snowflake.rest.runSql({...restConfig, statement}) And table expectedData | SERIAL_NUMBER | CUTTER_TYPE | | "MY_VECTOR" | "VECTOR" | And match response.status == "OK" And match response.data == expectedData
-
The
Background
part can be moved to akarate-config.js
file for better readability. -
The
Scenario
contains 3 steps :-
SQL
statement
declaration -
SQL execution :
snowflake.rest.runSql({...restConfig, statement})
-
Assertion on
response
, a JSON object like{"status":"OK", "data":[{"SERIAL_NUMBER": "MY_VECTOR", "CUTTER_TYPE": "VECTOR"}]}
-
Insert input - Run DBT - Select output
Feature: Demo Background: * json cliConfig = read('classpath:cli-config.json') * json snowflakeConfig = read('classpath:snowflake-config.json') * string jwt = snowflake.cli.generateJwt(cliConfig) * json restConfig = ({jwt, cliConfig, snowflakeConfig}) * string clientId = "😋_"+base.random.uuid() * def insert = (table, value) => "INSERT INTO "+table+"(CLIENT_ID, VALUE) VALUES ('"+clientId+"','"+value+"')" Scenario Outline: Burger Factory - <bread> + <vegetable> + <meat> = <output> Given table inserts | table | value | | "BREAD" | "<bread>" | | "VEGETABLE" | "<vegetable>" | | "MEAT" | "<meat>" | And json responses = karate.map(inserts, (row) => snowflake.rest.runSql({...restConfig, statement: insert(row.table, row.value)}).status) And match each responses == "OK" When json dbtResult = dbt.cli.run({}) And match dbtResult.status == "OK" And match dbtResult.output contains "Completed successfully" Then string select = "SELECT VALUE FROM BURGER WHERE CLIENT_ID='"+clientId+"'" And json response = snowflake.rest.runSql({...restConfig, statement: select }) And match response.data == [ { "VALUE" : "<output>" } ] Examples: | bread | vegetable | meat | output | | 🍞 | 🍅 | 🥩 | 🍔 | | 🍞 | 🍅 | 🍗 | 🍔 | | 🍞 | 🍅 | 🐟 | 🍔 | | 🍞 | 🥕 | 🥩 | 🍞 + 🥕 + 🥩 |
-
The
Background
part can also be moved to akarate-config.js
file for better readability. -
We have now a
Scenario Outline
= 1 scenario per row inExamples
part (tokens like<bread>
,<vegetable>
,<meat>
,<output>
will be replaced with emoji values). -
Every
Scenario
contains 3 parts :-
INSERT INTO
queries for burger ingredients :snowflake.rest.runSql({...restConfig, statement: insert(row.table, row.value)})
-
DBT
job execution :dbt.cli.run({})
-
SELECT
output table for assertions on a created burger :snowflake.rest.runSql({...restConfig, statement: select })
-
-
If you use more than 1 thread for your Karate run, for more isolation, you can clone your schema(s) in the
Background
part, and drop it (them) after the scenario. A full example is available in this feature with this karate-config.js configuration.
Insert a JSON-line file into a table
{"FOO":1,"BAR":"bar1"} {"FOO":2,"BAR":"bar2"}
* string fileAbsolutePath = karate.toAbsolutePath("myPath/file.json") * string tableName = "MY_TABLE" * json result = snowflake.cli.putJsonIntoTable({ fileAbsolutePath, tableName, cliConfig, snowflakeConfig }) * match result.status == "OK"
Conclusion
Karate Connect allows Lectra to conduct real integration tests of its data pipeline. It generates a very good feeback from our engineers. Current extensions (Snowflake, Rabbitmq, DBT, Kubernetes) should be completed with Apache Kafka and a generic JDBC connection soon. The tool is still in its early stages and requires enhancements (such as logging, performance, and missing features/extensions), but it’s worth testing ! We would appreciate any feedback you have.
More information:
-
See the README file : https://github.com/lectra-tech/karate-connect
-
Check examples in the demo project : https://github.com/mrebiai/karate-snowflake
Feel free to ⭐ the project, open issues or create pull requests for improvements.