The integration runtime (IR) is the compute infrastructure that Azure Data Factory uses to provideRegister Integration Runtime (Self-hosted) node with the specified Authentication KeyNoEnableRemoteAccess '
<port> ' ['<thumbprint> ']Enable remote access on the current node for setting up a High Availability Cluster and/or enabling setting of credentials directly against the self-hosted IR (without going through ADF service) using New-AzDataFactoryV2LinkedServiceEncryptedCredential cmdlet from a remote machine in same network.NoEnableRemoteAccessInContainer '<port> ' ['<thumbprint> ']Enable remote access to current node when the node is running in ContainerNoDisableRemoteAccessDisable remote access to current node. Remote access is needed for multi-node setup. The New-AzDataFactoryV2LinkedServiceEncryptedCredential PowerShell cmdlet still works even when remote access is disabled as long as it is executed on the same machine as the self-hosted IR node.NoKey '<AuthenticationKey> 'Overwrite/ update the previous Authentication Key. Please be careful as this can result in your previous self-hosted IR node going offline, if the key is of a new integration runtime.NoGenerateBackupFile '<filePath> ' '<password> 'Generate backup file for current node, the backup file includes the node key and data store credentialsNoImportBackupFile '<filePath> ' '<password> 'Restore the node from a backup fileNoRestartRestart the Integration Runtime (Self-hosted) Host ServiceNoStartStart the Integration Runtime (Self-hosted) Host ServiceNoStopStop Integration Runtime (Self-hosted) update serviceNoStartUpgradeServiceStart Integration Runtime (Self-hosted) update serviceNoStopUpgradeServiceStop Integration Runtime (Self-hosted) update serviceNoTurnOnAutoUpdateTurn on Integration Runtime (Self-hosted) auto updateNoTurnOffAutoUpdateTurn off Integration Runtime (Self-hosted) auto updateNoSwitchServiceAccount '<domainuser>' ['password']Set DIAHostService to run as a new account. Use empty password (') for system account or virtual accountNoLoglevel <logLevel> Set ETW log level (Off, Error, Verbose or All). Generally used by Microsoft support while debugging.No
-->
Command flow and data flow
When you move data between on-premises and the cloud, the activity uses a self-hosted integration runtime to transfer the data from an on-premises data source to the cloud and vice versa.
Here is a high-level data flow for the summary of steps for copying with a self-hosted IR:
A selection of our references: Proudly powered by WordPress Theme: Sydney by aThemes.
Considerations for using a self-hosted IR
Prerequisites
Installation best practices
You can install the self-hosted integration runtime by downloading an MSI setup package from the Microsoft Download Center. See Move data between on-premises and cloud article for step-by-step instructions.
Install and register self-hosted IR from the Download Center
High availability and scalability
A self-hosted integration runtime can be associated with multiple on-premises machines or Virtual Machines in Azure. These machines are called nodes. You can have up to four nodes associated with a self-hosted integration runtime. The benefits of having multiple nodes (on-premises machines with a gateway installed) for a logical gateway are:
You can associate multiple nodes by installing the self-hosted integration runtime software from the Download Center. Then, register it by using either of the authentication keys obtained from the New-AzDataFactoryV2IntegrationRuntimeKey cmdlet, as described in the tutorial.
Note
Free download boot camp for mac. You don't need to create new self-hosted integration runtime for associating each node. You can install the self-hosted integration runtime on another machine and register it by using the same authentication key.
Note
Before you add another node for high availability and scalability, ensure that the Remote access to intranet option is enabled on the first node (Microsoft Integration Runtime Configuration Manager > Settings > Remote access to intranet).
Scale considerationsScale out
When the available memory on the self-hosted IR is low and the CPU usage is high, adding a new node helps scale out the load across machines. If activities are failing because they're timing out or because the self-hosted IR node is offline, it helps if you add a node to the gateway.
Scale upDmg Intranet Page
When the available memory and CPU are not utilized well, but the execution of concurrent jobs is reaching the limit, you should scale up by increasing the number of concurrent jobs that can run on a node. You might also want to scale up when activities are timing out because the self-hosted IR is overloaded. As shown in the following image, you can increase the maximum capacity for a node:
TLS/SSL certificate requirements
Here are the requirements for the TLS/SSL certificate that is used for securing communications between integration runtime nodes:
Note
This certificate is used to encrypt ports on self-hosted IR node, used for node-to-node communication (for state synchronization which includes linked services' credentials synchronization across nodes) and while using PowerShell cmdlet for linked service credential setting from within local network. We suggest using this certificate if your private network environment is not secure or if you would like to secure the communication between nodes within your private network as well.Data movement in transit from self-hosted IR to other data stores always happens using encrypted channel, irrespective of this certificate set or not.
Create a shared self-hosted integration runtime in Azure Data Factory
You can reuse an existing self-hosted integration runtime infrastructure that you already set up in a data factory. This enables you to create a linked self-hosted integration runtime in a different data factory by referencing an existing self-hosted IR (shared).
For a twelve-minute introduction and demonstration of this feature, watch the following video:
Terminology
Methods to share a self-hosted integration runtime
To share a self-hosted integration runtime with multiple data factories, see this instruction for details.
Monitoring
Known limitations of self-hosted IR sharing
Notification area icons and notifications
If you move your cursor over the icon or message in the notification area, you can find details about the state of the self-hosted integration runtime.
Ports and firewall
There are two firewalls to consider: the corporate firewall running on the central router of the organization, and the Windows firewall configured as a daemon on the local machine where the self-hosted integration runtime is installed.
At the corporate firewall level, you need to configure the following domains and outbound ports:
At the Windows firewall level (machine level), these outbound ports are normally enabled. If not, you can configure the domains and ports accordingly on a self-hosted integration runtime machine.
Note
Based on your source and sinks, you might have to allow additional domains and outbound ports in your corporate firewall or Windows firewall.
For some cloud databases (for example, Azure SQL Database and Azure Data Lake), you might need to allow IP addresses of self-hosted integration runtime machines on their firewall configuration.
My Dmg EmployeeCopy data from a source to a sink
Ensure that the firewall rules are enabled properly on the corporate firewall, the Windows firewall on the self-hosted integration runtime machine, and the data store itself. Enabling these rules allows the self-hosted integration runtime to connect to both source and sink successfully. Enable rules for each data store that is involved in the copy operation.
For example, to copy from an on-premises data store to an Azure SQL Database sink or an Azure SQL Data Warehouse sink, take the following steps:
Note
If your firewall does not allow outbound port 1433, the self-hosted integration runtime can't access the Azure SQL database directly. In this case, you can use a staged copy to Azure SQL Database and Azure SQL Data Warehouse. In this scenario, you would require only HTTPS (port 443) for the data movement.
Proxy server considerations
If your corporate network environment uses a proxy server to access the internet, configure the self-hosted integration runtime to use appropriate proxy settings. You can set the proxy during the initial registration phase.
When configured, the self-hosted integration runtime uses the proxy server to connect to the cloud service, source/ destination (those using HTTP/ HTTPS protocol). This is Select Change link during initial setup. You see the proxy-setting dialog box.
There are three configuration options:
The integration runtime Host Service restarts automatically after you save the updated proxy settings.
After the self-hosted integration runtime has been successfully registered, if you want to view or update proxy settings, use Integration Runtime Configuration Manager.
![]()
Dmg Intranet Login
You can view and update the HTTP proxy by using the Configuration Manager tool.
Note
If you set up a proxy server with NTLM authentication, the integration runtime Host Service runs under the domain account. If you change the password for the domain account later, remember to update the configuration settings for the service and restart it accordingly. Due to this requirement, we suggest that you use a dedicated domain account to access the proxy server that does not require you to update the password frequently.
Configure proxy server settings
If you select the Use system proxy setting for the HTTP proxy, the self-hosted integration runtime uses the proxy setting in diahost.exe.config and diawp.exe.config. If no proxy is specified in diahost.exe.config and diawp.exe.config, the self-hosted integration runtime connects to the cloud service directly without going through proxy. The following procedure provides instructions for updating the diahost.exe.config file:
Important
Don't forget to update both diahost.exe.config and diawp.exe.config.
You also need to make sure that Microsoft Azure is in your company’s allow list. You can download the list of valid Microsoft Azure IP addresses from the Microsoft Download Center.
Dmv Intranet NyPossible symptoms for firewall and proxy server-related issues
If you encounter errors similar to the following ones, it's likely due to improper configuration of the firewall or proxy server, which blocks the self-hosted integration runtime from connecting to Data Factory to authenticate itself. To ensure that your firewall and proxy server are properly configured, refer to the previous section.
Enabling remote access from an intranetDmg Pay My Bill
If you use PowerShell to encrypt credentials from another machine (in the network) other than where the self-hosted integration runtime is installed, you can enable the Remote Access from Intranet option.If you run PowerShell to encrypt credentials on the same machine where the self-hosted integration runtime is installed, you can't enable Remote Access from Intranet.
You should enable Remote Access from Intranet before you add another node for high availability and scalability.
During self-hosted integration runtime setup (version 3.3.xxxx.x later), by default, the self-hosted integration runtime installation disables Remote Access from Intranet on the self-hosted integration runtime machine.
If you're using a third-party firewall, you can manually open port 8060 (or the user-configured port). If you have a firewall problem while setting up the self-hosted integration runtime, try using the following command to install the self-hosted integration runtime without configuring the firewall:
If you choose not to open port 8060 on the self-hosted integration runtime machine, use mechanisms other than the Setting Credentials application to configure data store credentials. For example, you can use the New-AzDataFactoryV2LinkedServiceEncryptCredential PowerShell cmdlet.
Next steps
See the following tutorial for step-by-step instructions: Tutorial: Copy on-premises data to cloud.
If you have not registered for a membership,
|
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |