Skip to content

Commit 81b319c

Browse files
itsmekumariitsmekumari
authored
e2e_mssql replication tests (#248)
1 parent 9b2e1d2 commit 81b319c

File tree

18 files changed

+1014
-1
lines changed

18 files changed

+1014
-1
lines changed

‎.github/workflows/e2e.yml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ jobs:
4040
)
4141
strategy:
4242
matrix:
43-
module: [mysql-delta-plugins]
43+
module: [mysql-delta-plugins, sqlserver-delta-plugins]
4444
fail-fast: false
4545

4646
steps:
@@ -83,6 +83,10 @@ jobs:
8383
MYSQL_USERNAME:cdapio-github-builds/MYSQL_USERNAME
8484
MYSQL_PASSWORD:cdapio-github-builds/MYSQL_PASSWORD
8585
MYSQL_PORT:cdapio-github-builds/MYSQL_PORT
86+
MSSQL_HOST:cdapio-github-builds/MSSQL_HOST
87+
MSSQL_USERNAME:cdapio-github-builds/MSSQL_USERNAME
88+
MSSQL_PASSWORD:cdapio-github-builds/MSSQL_PASSWORD
89+
MSSQL_PORT:cdapio-github-builds/MSSQL_PORT
8690
8791
- name: Run required e2e tests
8892
if: github.event_name != 'workflow_dispatch' && github.event_name != 'push' && steps.filter.outputs.e2e-test == 'false'
@@ -92,6 +96,10 @@ jobs:
9296
MYSQL_USERNAME: ${{ steps.secrets.outputs.MYSQL_USERNAME }}
9397
MYSQL_PASSWORD: ${{ steps.secrets.outputs.MYSQL_PASSWORD }}
9498
MYSQL_PORT: ${{ steps.secrets.outputs.MYSQL_PORT }}
99+
MSSQL_HOST: ${{ steps.secrets.outputs.MSSQL_HOST }}
100+
MSSQL_USERNAME: ${{ steps.secrets.outputs.MSSQL_USERNAME }}
101+
MSSQL_PASSWORD: ${{ steps.secrets.outputs.MSSQL_PASSWORD }}
102+
MSSQL_PORT: ${{ steps.secrets.outputs.MSSQL_PORT }}
95103

96104
- name: Run all e2e tests
97105
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push' || steps.filter.outputs.e2e-test == 'true'
@@ -101,6 +109,10 @@ jobs:
101109
MYSQL_USERNAME: ${{ steps.secrets.outputs.MYSQL_USERNAME }}
102110
MYSQL_PASSWORD: ${{ steps.secrets.outputs.MYSQL_PASSWORD }}
103111
MYSQL_PORT: ${{ steps.secrets.outputs.MYSQL_PORT }}
112+
MSSQL_HOST: ${{ steps.secrets.outputs.MSSQL_HOST }}
113+
MSSQL_USERNAME: ${{ steps.secrets.outputs.MSSQL_USERNAME }}
114+
MSSQL_PASSWORD: ${{ steps.secrets.outputs.MSSQL_PASSWORD }}
115+
MSSQL_PORT: ${{ steps.secrets.outputs.MSSQL_PORT }}
104116

105117
- name: Upload report
106118
uses: actions/upload-artifact@v3

‎sqlserver-delta-plugins/src/e2e-test/features/mssql/DesignTimeValidation.feature‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
#
2+
# Copyright © 2023 Cask Data, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
5+
# use this file except in compliance with the License. You may obtain a copy of
6+
# the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
12+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13+
# License for the specific language governing permissions and limitations under
14+
# the License.
15+
#
16+
17+
@Mssql
18+
Feature: Mssql - Verify Mssql source plugin design time validation scenarios
19+
20+
@Required
21+
Scenario: To verify validation message when user provides invalid Host
22+
Given Open DataFusion Project with replication to configure replication job
23+
When Enter input plugin property: "name" with pipelineName
24+
And Click on the "Next" button in replication to navigate
25+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
26+
Then Replace input plugin property: "host" with value: "mssqlInvalidHost" for Credentials and Authorization related fields
27+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
28+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
29+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
30+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
31+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
32+
And Click on the "Next" button in replication to navigate
33+
Then Replace input plugin property: "project" with value: "projectId"
34+
Then Enter input plugin property: "datasetName" with value: "dataset"
35+
And Click on the "Next" button in replication to navigate
36+
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidMssqlHost"
37+
38+
@Required
39+
Scenario: To verify validation message when user provides invalid Port
40+
Given Open DataFusion Project with replication to configure replication job
41+
When Enter input plugin property: "name" with pipelineName
42+
And Click on the "Next" button in replication to navigate
43+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
44+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
45+
Then Replace input plugin property: "port" with value: "mssqlInvalidPort" for Credentials and Authorization related fields
46+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
47+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
48+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
49+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
50+
And Click on the "Next" button in replication to navigate
51+
Then Replace input plugin property: "project" with value: "projectId"
52+
Then Enter input plugin property: "datasetName" with value: "dataset"
53+
And Click on the "Next" button in replication to navigate
54+
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidMssqlPort"
55+
56+
Scenario: To verify validation message when user provides invalid Database Name
57+
Given Open DataFusion Project with replication to configure replication job
58+
When Enter input plugin property: "name" with pipelineName
59+
And Click on the "Next" button in replication to navigate
60+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
61+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
62+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
63+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
64+
Then Replace input plugin property: "database" with value: "mssqlInvalidDatabaseName"
65+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
66+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
67+
And Click on the "Next" button in replication to navigate
68+
Then Replace input plugin property: "project" with value: "projectId"
69+
Then Enter input plugin property: "datasetName" with value: "dataset"
70+
And Click on the "Next" button in replication to navigate
71+
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidMssqlDatabase"
72+
73+
@Required
74+
Scenario: To verify validation message when user provides invalid user
75+
Given Open DataFusion Project with replication to configure replication job
76+
When Enter input plugin property: "name" with pipelineName
77+
And Click on the "Next" button in replication to navigate
78+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
79+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
80+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
81+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
82+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
83+
Then Replace input plugin property: "user" with value: "mssqlInvalidUser" for Credentials and Authorization related fields
84+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
85+
And Click on the "Next" button in replication to navigate
86+
Then Replace input plugin property: "project" with value: "projectId"
87+
Then Enter input plugin property: "datasetName" with value: "dataset"
88+
And Click on the "Next" button in replication to navigate
89+
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidMssqlUser"
90+
91+
Scenario: To verify validation message when user provides invalid password
92+
Given Open DataFusion Project with replication to configure replication job
93+
When Enter input plugin property: "name" with pipelineName
94+
And Click on the "Next" button in replication to navigate
95+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
96+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
97+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
98+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
99+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
100+
Then Replace input plugin property: "user" with value: "mssqlInvalidUser" for Credentials and Authorization related fields
101+
Then Replace input plugin property: "password" with value: "mssqlInvalidPassword" for Credentials and Authorization related fields
102+
And Click on the "Next" button in replication to navigate
103+
Then Replace input plugin property: "project" with value: "projectId"
104+
Then Enter input plugin property: "datasetName" with value: "dataset"
105+
And Click on the "Next" button in replication to navigate
106+
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidMssqlPassword"
107+
108+
Scenario: To verify validation message when macro enabled for password field
109+
Given Open DataFusion Project with replication to configure replication job
110+
When Enter input plugin property: "name" with pipelineName
111+
And Click on the "Next" button in replication to navigate
112+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
113+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
114+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
115+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
116+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
117+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
118+
Then Click on the Macro button of Property: "password" and set the value to: "mssqlPassword"
119+
And Click on the "Next" button in replication to navigate
120+
Then Replace input plugin property: "project" with value: "projectId"
121+
Then Enter input plugin property: "datasetName" with value: "dataset"
122+
And Click on the "Next" button in replication to navigate
123+
Then Verify that the Plugin is displaying an error message: "errorMessageMacroPassword"
124+
125+
Scenario: To verify validation message when macro enabled for service account key field
126+
Given Open DataFusion Project with replication to configure replication job
127+
When Enter input plugin property: "name" with pipelineName
128+
And Click on the "Next" button in replication to navigate
129+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
130+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
131+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
132+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
133+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
134+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
135+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
136+
And Click on the "Next" button in replication to navigate
137+
Then Replace input plugin property: "project" with value: "projectId"
138+
Then Click on the Macro button of Property: "serviceAccountKey" and set the value to: "ServiceAccountKey"
139+
Then Enter input plugin property: "datasetName" with value: "dataset"
140+
And Click on the "Next" button in replication to navigate
141+
Then Verify that the Plugin is displaying an error message: "errorMessageMacroServiceAccountKey"

‎sqlserver-delta-plugins/src/e2e-test/features/mssql/RunTime.feature‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
#
2+
# Copyright © 2023 Cask Data, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
5+
# use this file except in compliance with the License. You may obtain a copy of
6+
# the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
12+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13+
# License for the specific language governing permissions and limitations under
14+
# the License.
15+
#
16+
17+
@Mssql @Required
18+
Feature: Mssql - Verify Mssql source data transfer to BigQuery
19+
20+
@MSSQL_SOURCE @MSSQL_DELETE @BQ_SINK_TEST
21+
Scenario: To verify replication of snapshot and cdc data from MsSQL to BigQuery successfully
22+
Given Open DataFusion Project with replication to configure replication job
23+
When Enter input plugin property: "name" with pipelineName
24+
And Click on the "Next" button in replication to navigate
25+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
26+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
27+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
28+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
29+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
30+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
31+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
32+
And Click on the "Next" button in replication to navigate
33+
Then Replace input plugin property: "project" with value: "projectId"
34+
Then Enter input plugin property: "datasetName" with value: "dataset"
35+
And Click on the "Next" button in replication to navigate
36+
Then Select the source table if available
37+
And Click on the "Next" button in replication to navigate
38+
Then Wait till the Configure Advanced Properties page is loaded in replication
39+
Then Click on the "Next" button in replication to navigate
40+
Then Wait till the Review Assessment page is loaded in replication
41+
Then Click on the "Next" button in replication to navigate
42+
Then Deploy the replication pipeline
43+
And Run the replication Pipeline
44+
Then Open the Advanced logs
45+
And Wait till pipeline is in running state and check if no errors occurred
46+
Then Validate the values of records transferred to target BigQuery table is equal to the values from MsSQL source Table
47+
And Run insert, update and delete CDC events on source table
48+
And Wait till CDC events are reflected in BQ
49+
Then Validate the values of records transferred to target BigQuery table is equal to the values from MsSQL source Table
50+
And Open and capture logs
51+
Then Close the replication pipeline logs and stop the pipeline
52+
53+
@MSSQL_SOURCE @MSSQL_DELETE @BQ_SINK_TEST
54+
Scenario: To verify replication of snapshot data from Mssql to BigQuery successfully
55+
Given Open DataFusion Project with replication to configure replication job
56+
When Enter input plugin property: "name" with pipelineName
57+
And Click on the "Next" button in replication to navigate
58+
And Select Source plugin: "Microsoft SQLServer" from the replication plugins list
59+
Then Replace input plugin property: "host" with value: "mssqlHost" for Credentials and Authorization related fields
60+
Then Replace input plugin property: "port" with value: "mssqlPort" for Credentials and Authorization related fields
61+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "mssqlDriverName"
62+
Then Replace input plugin property: "database" with value: "mssqlDatabaseName"
63+
Then Replace input plugin property: "user" with value: "mssqlUsername" for Credentials and Authorization related fields
64+
Then Replace input plugin property: "password" with value: "mssqlPassword" for Credentials and Authorization related fields
65+
And Click on the "Next" button in replication to navigate
66+
Then Replace input plugin property: "project" with value: "projectId"
67+
Then Enter input plugin property: "datasetName" with value: "dataset"
68+
Then Click on the "Next" button in replication to navigate
69+
Then Select the source table if available
70+
Then Click on the "Next" button in replication to navigate
71+
Then Wait till the Configure Advanced Properties page is loaded in replication
72+
Then Click on the "Next" button in replication to navigate
73+
Then Wait till the Review Assessment page is loaded in replication
74+
Then Click on the "Next" button in replication to navigate
75+
Then Deploy the replication pipeline
76+
And Run the replication Pipeline
77+
Then Open the Advanced logs
78+
And Wait till pipeline is in running state and check if no errors occurred
79+
Then Validate the values of records transferred to target BigQuery table is equal to the values from MsSQL source Table
80+
And Open and capture logs
81+
Then Close the replication pipeline logs and stop the pipeline

0 commit comments

Comments
 (0)