Incremental Extract in Prep

I'm having a lot of difficulties creating an incremental extract in Prep; I'm uncertain what I'm doing incorrectly and could use some expert advice.

The goal is to create a 3-month* audit log archive that will grow as new records are added (daily).

I'm linking the flow to our company's audit log, however the log is so massive the flow times out.

To resolve this I decided to partition the months: create prep flows that would generate "bite sized" monthly chunks, then execute a flow that would combine them together to create a single file (called the "3 Month Archive"). This, naturally, added a new column called "Table Names" which I did not delete.

Next, I used the same Prep logic that created the various month buckets, but pointed it to the 3 Month Archive** and removed the date filter. (Do I need to remove the date filter if I know this is running regularly? Is it faster if I leave the date filter in?)

**Yes, this wasn't intuitive to me: I'm pointing to the same file I'm going to be writing to-- but I guess that's how Increment Refresh works?

I implemented Incremental Refresh using the DateTime field (only one data source has this).

In the output table specifics I set the full refresh to APPEND and Incremental refresh to APPEND.

When I ran the flow, I chose Incremental Refresh rather than Full Refresh.

The flow stops almost immediately due to an "Internal Error" when I choose Incremental Refresh for the Extract Task Type.

If I try to do a full refresh (default) it times out.

Any thoughts on this?

Am I looking at this the wrong way? Knowing we have 2 months ago,1 month ago, should I simply:

• Refresh the past months on a monthly basis
• Refresh the current month on a daily basis
• On a daily basis, combine them all together as a new extract?

I was hoping Incremental extract would be very fast.

We are using the DM package, so I can schedule task order. Any feedback appreciated!

Michael Hesser (Tableau Forum Ambassador)

If this response has answered your question, kindly click "Best Answer"