You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This document focuses on the best practices to use existing SSIS packages to work with Data warehouse in Fabric platform.
19
+
This article focuses on the best practices to use existing SSIS packages to work with Data warehouse in Fabric platform.
19
20
20
21
## Introduction
21
22
22
-
****Microsoft Fabric**** is a comprehensive analytics platform that covers every aspect of an organization's data estate. One of its key experiences is Fabric Data Warehouse, which serves as a simplified SaaS solution for a fully transactional warehouse. It stores data in OneLake using an open format called Delta Parquet, ensuring that data can be accessed by other experiences within Fabric and other client applications that connect using SQL drivers.
23
+
Microsoft Fabric is a comprehensive analytics platform that covers every aspect of an organization's data estate. One of its key experiences is Fabric Data Warehouse, which serves as a simplified SaaS solution for a fully transactional warehouse. It stores data in OneLake using an open format called Delta Parquet, ensuring that data can be accessed by other experiences within Fabric and other client applications that connect using SQL drivers.
23
24
24
-
****Microsoft Fabric****, as an analytics platform, exclusively supports authentication through ****Microsoft Entra ID**** for users and Service Principals (SPNs). This deliberate choice ensures centralized and identity-based security, aligning with modern security practices. So, SQL authentication and other authentication methods aren't supported in Fabric Data Warehouse within the Fabric ecosystem.
25
+
As an analytics platform, Microsoft Fabric exclusively supports authentication through Microsoft Entra ID for users and service principals (SPNs). This deliberate choice ensures centralized and identity-based security, aligning with modern security practices. So, SQL authentication and other authentication methods aren't supported in Fabric Data Warehouse within the Fabric ecosystem.
25
26
26
27
## Integration with Fabric Data Warehouse
27
-
Microsoft SQL Server Integration Services (SSIS) is a component of the Microsoft SQL Server database that is an ETL solution. SSIS is widely used by enterprise customers to perform ETL on premises by many customers.
28
28
29
-
Two key modifications are required in SSIS package to work seamlessly with Fabric Data Warehouse, outlined as follows.
29
+
Microsoft SQL Server Integration Services (SSIS) is a component of the Microsoft SQL Server database that's an ETL solution. Many enterprise customers widely use SSIS to perform on-premises ETL.
30
+
31
+
To work seamlessly with Fabric Data Warehouse, you need to make two key modifications to your SSIS package.
30
32
31
33
### Authentication
32
-
If you're using SQL Authentication or Windows Authentication, reconfigure it to utilize Microsoft Entra ID User or Service Principal Name (SPN). Keep in mind that if you're using a User account, multifactor authentication (MFA) must be disabled, as SSIS doesn't support pop-up prompts. It also needs respective drivers as mentioned below:
33
34
34
-
****To use [OLEDB connection manager](../connection-manager/ole-db-connection-manager.md)****:
35
-
- Install [OLE DB Driver for SQL Server](../../connect/oledb/features/using-azure-active-directory.md) version that supports Microsoft Entra ID
36
-
- Set Authentication to ****ActiveDirectoryServicePrincipal**** or ****ActiveDirectoryPassword****.
35
+
If you're using SQL Authentication or Windows Authentication, reconfigure it to use Microsoft Entra ID User or Service Principal Name (SPN). If you use a user account, disable multifactor authentication (MFA), because SSIS doesn't support pop-up prompts. You also need the respective drivers as mentioned in the following sections:
36
+
37
+
To use [OLEDB connection manager](../connection-manager/ole-db-connection-manager.md):
38
+
39
+
- Install [Use Microsoft Entra ID](../../connect/oledb/features/using-azure-active-directory.md) version that supports Microsoft Entra ID.
40
+
41
+
- Set Authentication to `ActiveDirectoryServicePrincipal` or `ActiveDirectoryPassword`.
42
+
37
43
- OLEDB only works for [Execute SQL Task](../control-flow/execute-sql-task.md), doesn't work for [OLE DB Destination](../data-flow/ole-db-destination.md).
38
-
:::image type="content" border="false" source="media/ole-db-connection-1.png" alt-text="Screenshot of oledb connection manager part 1." lightbox="media/ole-db-connection-1.png":::
39
-
:::image type="content" source="media/ole-db-connection-2.png" alt-text="Screenshot of oledb connection manager part 2." lightbox="media/ole-db-connection-2.png":::
40
-
41
-
****To use ADO.NET connection manager****:
42
-
- Use Microsoft OLE DB provider for SQL Server for [.NET Framework Data Provider for OLE DB](/dotnet/framework/data/adonet/data-providers).
43
-
- Set Authentication to ****ActiveDirectoryServicePrincipal**** or ****ActiveDirectoryPassword****.
44
-
:::image type="content" source="media/ado-net-connection.png" alt-text="Screenshot of ado connection manager part 1." lightbox="media/ado-net-connection.png":::
45
-
46
-
### File ingestion
47
-
The ****Fabric Data Warehouse**** recommends utilizing the native T-SQL command 'COPY INTO' for efficient data insertion into the warehouse. So, any DFT operations that currently rely on ****Fast Insert Mode**** or ****BCP IN**** scripts should be replaced with the ****COPY INTO**** statement by utilizing [Execute SQL Task](../control-flow/execute-sql-task.md).
44
+
45
+
:::image type="content" source="media/ole-db-connection-1.png" alt-text="Screenshot of oledb connection manager part 1.":::
46
+
47
+
:::image type="content" source="media/ole-db-connection-2.png" alt-text="Screenshot of oledb connection manager part 2." lightbox="media/ole-db-connection-2.png":::
48
+
49
+
To use ADO.NET connection manager:
50
+
51
+
- Use Microsoft OLE DB provider for SQL Server for [.NET Framework Data Provider for OLE DB](/dotnet/framework/data/adonet/data-providers).
52
+
53
+
- Set Authentication to `ActiveDirectoryServicePrincipal` or `ActiveDirectoryPassword`.
54
+
55
+
:::image type="content" source="media/ado-net-connection.png" alt-text="Screenshot of ado connection manager part 1." lightbox="media/ado-net-connection.png":::
56
+
57
+
### File ingestion
58
+
59
+
You should use the native `COPY INTO` T-SQL command for efficient data insertion into your warehouse in Microsoft Fabric. Replace any DFT operations that currently rely on **Fast Insert Mode** or `BCP IN` scripts with the `COPY INTO` statement by using [Execute SQL Task](../control-flow/execute-sql-task.md).
48
60
49
61
### SSIS writing data into Data Warehouse in Fabric
50
62
51
-
It's a common ETL scenario where data is read from different sources like transactional databases, network file shares, local/network etc., perform transformation steps and write back to a designated DW like a SQL server, synapse dedicated pool or any other SQL compliant data store (like shown below in the diagram).
63
+
In common ETL scenarios, you read data from different sources like transactional databases, network file shares, local or network locations. You can perform transformation steps and write the data back to a designated data warehouse like a SQL server, Synapse dedicated pool, or any other SQL compliant data store (as shown in the following diagram).
64
+
65
+
:::image type="content" source="media/etl-data-warehouse-destination.png" alt-text="Diagram of etl data warehouse as destination.":::
66
+
67
+
To make the same SSIS package write to Fabric Data Warehouse, first update the authentication to Microsoft Entra ID based if it's not already used. Second, temporarily stage the data in an ADLS Gen2. Then pass the path to the COPY INTO command in [Execute SQL Task](../control-flow/execute-sql-task.md).
68
+
69
+
[Flexible File Destination](../data-flow/flexible-file-destination.md) component enables an SSIS package to write data to [Azure Data Lake Storage Gen2 (ADLS Gen2)](/azure/storage/blobs/data-lake-storage-introduction). Inside Data Flow task, after loading and transformation, add a [Flexible File Destination](../data-flow/flexible-file-destination.md), in which you can define destination file name and location in ADLS Gen2.
52
70
53
-
:::image type="content" border="false" source="media/etl-data-warehouse-destination.png" alt-text="Diagram of etl data warehouse as destination.":::
71
+
:::image type="content" source="media/flexible-file-1.png" alt-text="Screenshot of Flexible file destination part 1.":::
54
72
55
-
In order to make same SSIS package to write to Fabric Data Warehouse, First, update the authentication to Microsoft Entra ID based if not already used. Second, temporarily stage the data in an ADLS Gen2. Then pass the path to COPY INTO command in [Execute SQL Task](../control-flow/execute-sql-task.md).
56
-
73
+
:::image type="content" source="media/flexible-file-2.png" alt-text="Screenshot of Flexible file destination part 2.":::
57
74
58
-
[Flexible File Destination](../data-flow/flexible-file-destination.md) component enables an SSIS package to write data to [Azure Data Lake Storage Gen2 (ADLS Gen2)](/azure/storage/blobs/data-lake-storage-introduction). Inside Data Flow task, after loading and transformation, add a [Flexible File Destination](../data-flow/flexible-file-destination.md), in which you can define destination file name and location in ADLS Gen2.
75
+
:::image type="content" source="media/flexible-file-3.png" alt-text="Screenshot of Flexible filedestination part 3.":::
59
76
60
-
:::image type="content" source="media/flexible-file-1.png" alt-text="Screenshot of Flexible file destination part 1." lightbox="media/flexible-file-1.png":::
61
-
:::image type="content" source="media/flexible-file-2.png" alt-text="Screenshot of Flexible file destination part 2." lightbox="media/flexible-file-2.png":::
62
-
:::image type="content" source="media/flexible-file-3.png" alt-text="Screenshot of Flexible file destination part 3." lightbox="media/flexible-file-3.png":::
77
+
You can ingest data landed in Azure Data Lake Storage (ADLS) Gen2 into Warehouse with the `COPY` statement directly via [Execute SQL Task](../control-flow/execute-sql-task.md).
63
78
64
-
Data landed in Azure Data Lake Storage (ADLS) Gen2 can be ingested into Warehouse using COPY statement directly via [Execute SQL Task](../control-flow/execute-sql-task.md).
79
+
For example (replace `<storage_account>`, `<storage_account_key>` and `account_key` with valid values):
COPY INTO table_name FROM'https://<storage_account>.dfs.core.windows.net/<folder>/'
83
+
WITH (FILE_TYPE ='CSV',
84
+
CREDENTIAL = (IDENTITY ='<storage_account_key>',
85
+
SECRET ='<account_key>'),
86
+
FIELDQUOTE ='"',
87
+
FIELDTERMINATOR =',',
88
+
ROWTERMINATOR ='0x0A',
89
+
ENCODING ='UTF8'
90
+
);
78
91
```
79
-
:::image type="content" source="media/execute-sql-task.png" alt-text="Screenshot of Execute sql task." lightbox="media/execute-sql-task.png" :::
80
92
81
-
More detail instructions refer to [Ingest data into your Warehouse using the COPY statement](/fabric/data-warehouse/ingest-data-copy).
93
+
:::image type="content" source="media/execute-sql-task.png" alt-text="Screenshot of Execute SQL Task." lightbox="media/execute-sql-task.png":::
82
94
83
-
### Known limitations
95
+
For more detailed instructions, see [Ingest data into your Warehouse using the COPY statement](/fabric/data-warehouse/ingest-data-copy).
84
96
85
-
Fabric data Warehouse supports a subset of T-SQL data types and not all T-SQL all commands are currently supported. Your packages might be failed due to unsupported features. For details, please check [Data types in Warehouse](/fabric/data-warehouse/data-types?branch=main) and [T-SQL surface area](/fabric/data-warehouse/tsql-surface-area).
97
+
## Limitations
86
98
87
-
### References
88
-
[T-SQL surface area - Microsoft Fabric | Microsoft Learn](/fabric/data-warehouse/tsql-surface-area)
99
+
Fabric data Warehouse supports a subset of T-SQL data types and not all T-SQL commands are currently supported. Your packages might fail due to unsupported features. For details, check [Data types in Warehouse](/fabric/data-warehouse/data-types) and [T-SQL surface area in Fabric Data Warehouse](/fabric/data-warehouse/tsql-surface-area).
89
100
90
-
[Options to get data into the Lakehouse - Microsoft Fabric | Microsoft Learn](/fabric/data-engineering/load-data-lakehouse)
101
+
### Related content
91
102
92
-
[Ingesting data into the warehouse - Microsoft Fabric | Microsoft Learn](/fabric/data-warehouse/ingest-data)
103
+
-[T-SQL surface area in Fabric Data Warehouse](/fabric/data-warehouse/tsql-surface-area)
104
+
-[Options to get data into the Fabric Lakehouse](/fabric/data-engineering/load-data-lakehouse)
105
+
-[Ingest data into the Warehouse](/fabric/data-warehouse/ingest-data)
0 commit comments