Issue with databricks

Hi,

I am using XlsxWriter on Databricks to write an Excel file on an Azure blob but I have the following issue : FileCreateError: [Errno 95] Operation not supported.

I am using Python version 3.7.3 and XlsxWriter 1.2.8.

Here is some code that demonstrates the problem:

import xlsxwriter
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))

writer = pd.ExcelWriter("output.xlsx", engine='xlsxwriter')

df.to_excel(writer, sheet_name='output_1')

writer.save()

[pfernandez-sanofi]

I managed to solve it by saving the file locally in Databricks, then I copy it to the desired mount.

import datetime
import shutil

now_= datetime.datetime.now()
timestamp_ = datetime.datetime.timestamp(now_)
tmp_path = f'temp_{timestamp_}.xlsx' # I save it with the timestamp to avoid errors
writer = ExcelWriter(tmp_path)
real_path = f'{YOURPATH}_filename.xlsx'
df.to_excel(writer,'sheet_name')
writer.save()
shutil.copy(tmp_path, real_path) # Copy the temp file to mount 

[jmcnamara]

Resolved

Similar to the suggestion from @pfernandez-sanofi above the Databricks documentation now recommends performing the operation on a local disk and then copying the output XlsxWriter file to dbfs.

From the docs https://docs.databricks.com/data/databricks-file-system.html#local-file-api-limitations:

Local file API limitations

FUSE V2 (default for Databricks Runtime 6.x and 7.x) Does not support random writes. For workloads that require random writes, perform the I/O on local disk first and then copy the result to /dbfs. For example:

import xlsxwriter
from shutil import copyfile

workbook = xlsxwriter.Workbook('/local_disk0/tmp/excel.xlsx')
worksheet = workbook.add_worksheet()
worksheet.write(0, 0, "Key")
worksheet.write(0, 1, "Value")
workbook.close()

copyfile('/local_disk0/tmp/excel.xlsx', '/dbfs/tmp/excel.xlsx')