Proxy Support in Ingest Client

Is your feature request related to a problem? Please describe. I’m frustrated when we want to send data to ADX from firewalled network. https_proxy environment variable works, but because it is global settings of python process, we need to use proxy as parameter.

Recent change https://github.com/Azure/azure-kusto-python/pull/360 was affected only query side but in my understanding, still we cannot use proxy in IngestClient

Therefore, my script stopped because it cannot reach blob url.

Describe the solution you’d like

In terms of implementation, I don’t have good idea to solve. (because of the complexity of implementation) but I want to use proxy settings without environment variable setting. And I also find a clue to solve, hopefuly my Additional context in later is helpful

Describe alternatives you’ve considered None

Additional context 1) Here is my code snipets to upload data with QueuedIngestClient:ingest_from_file.

from azure.kusto.data import KustoConnectionStringBuilder
from azure.kusto.ingest import QueuedIngestClient, IngestionProperties, FileDescriptor
from azure.kusto.data.data_format import DataFormat

def postdata(self, dictdata, database, table, json_ingestion_mapping):
        ingestion_props = IngestionProperties(database=database,
                                              table=table,
                                              data_format=DataFormat.MULTIJSON,
                                              ingestion_mapping_reference=json_ingestion_mapping)
        kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(self.cluster_url,
                                                                                    self.client_id,
                                                                                    self.client_secret,
                                                                                    self.authority_id)
        client = QueuedIngestClient(kcsb)
        client._resource_manager._kusto_client.set_proxy(proxy_url=self.proxy)

        with tempfile.NamedTemporaryFile("w") as tp:
            tp.write(json.dumps(dictdata))
            tp.seek(0)
            file_descriptor = FileDescriptor(tp.name)
            client.ingest_from_file(file_descriptor, ingestion_properties=ingestion_props)

Thanks to https://github.com/Azure/azure-kusto-python/pull/360, we can set proxy to ksuto_client in resource_manager. (still need access non-public parameters though)

client._resource_manager._kusto_client.set_proxy(proxy_url=self.proxy)

But, in my understanding, because following implementation creates blob client & upload data without using any injected config, we cannot injected proxy config. https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-ingest/azure/kusto/ingest/ingest_client.py#L121

            blob_service = BlobServiceClient(random_container.account_uri)
            blob_client = blob_service.get_blob_client(container=random_container.object_name, blob=blob_name)
            blob_client.upload_blob(data=stream)

As far as I confirmed following official examples, we can add proxy settings to first line in above code.

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/samples/blob_samples_proxy_configuration.py#L51

service_client = BlobServiceClient.from_connection_string(connection_string, proxies=proxies)

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/azure/storage/blob/_blob_service_client.py#L180

    @classmethod
    def from_connection_string(
            cls,  # type: Type[ClassType]
            conn_str,  # type: str
            credential=None,  # type: Optional[Any]
            **kwargs  # type: Any
        ):  # type: (...) -> ClassType
        account_url, secondary, credential = parse_connection_str(conn_str, credential, 'blob')
        if 'secondary_hostname' not in kwargs:
            kwargs['secondary_hostname'] = secondary
        return cls(account_url, credential=credential, **kwargs)

Actually, I didn’t expect this function uses blob even if we call ingest_from_file. I assumed it uploads data to ADX directly. I also propose to add more doc to the function.

[AsafMah]

Solved in version 3.1.0

Feel free to re-open if there are any issues

[AsafMah]

1. I am working on a solution
2. Queued client always uploads to a blob (in ADX’s storage), if you want to ingest a file “directly”, you can use StreamingIngest or ManagedStreamingIngest