I tried reducing the flow steps by combining few clean steps together, but that doesn't solve the issue. Also I find no answer available online for this issue. can someone help guiding me here to resolve this issue?
6 answers
Hi @GARVIT PANT ,
I found the solution to this problem.
It turns out that a pivot in my Prep flow was generating 600 fields. MongoDB operates a two-stage combination of "$unionWith + $project" for the pivot operation, effectively doubling the number of stages to 1200. This exceeds MongoDB's pipeline stage limit of 1000, resulting in the error.
Solution:
- Tried splitting the 501 fields into two (300+300) - Excluded and included the respective fields at the column level.
- Pivoted them separately
- UNION'ed them at the later stage.
