Hi Guys,
I have huge amount of csv files under different projects folder. What I want to do is to set up a project filter first, and then based on this project Python code will scan all the files under this folder. Then I can use this return file lists as another filter. After that, python will open specific csv files based on these two steps filter. The reason to do that is that I don't want to load all the data into tableau one time, it's huge amount of data as I mentioned.
I can easily write code in the Python Jupyter Notebook. But it's not easy to complete this in Tableau for me. Hopefully, I can get the idea from you guys. Thank you so much.
My code will be like this:
#Code to read all the files under a filtered project
⌗This code is okay in Jupyter notebook, but has error in the tableau, I cannot figure it out why it has error
script_str(
"
import os
Proj = _arg1 # Project filter
Folder = r"C:\Samples\" + Proj
csvfiles = [f for f in oslist(Folder) if f.endswith(".csv")
datefile = [f[:8] for f in csvfiles]
return datefiles
",
attr([Project]))
#Code to read a specific file based on csvfiles filter
#The reason to return a value list, because I don't know how to return the whole dataset
#If we can return a dataframe directly, it will great. Otherwise, I need to repeat this step many many time to get the value I want
script_str(
"
import pandas as pd
import os
Proj = _arg1 # Project filter
DT = _arg2 # The csv file filter
Folder = r"C:\Samples\" + Proj
csvfiles = [f for f in oslist(Folder) if f.endswith(".csv")
for item in csvfiles:
if DT in item:
df = pd.read_csv(Folder + "//" + item)
return df["Value"].tolist()
",
attr([Project]), attr([Date]))
