I'm working on using the Tableau Hyper API to load data from pandas Dataframes to Hyper directly and am trying to see if the data can be loaded as int64's representing timestamps. It looks like we can get up to a 6x speedup by adding int64's directly instead of converting through year/month/day/hr/minute/sec/us.
Currently it looks like the the int64 that is used by pantab (see Further Performance Optimizavtion? · Issue #41 · innobi/pantab · GitHub ) is Julien date derived, so I am able to convert pandas Timestamp columns to int64 plus an offset of 2440588*86400*1000000, representing [the day diff between the Julien epoch starting at midnight and the unix epoch of 1970-01-01] * 86400 (seconds/day) * 1MM microseconds / second.
Can someone in the Hyper API team verify the format? Also would be nice to know the values for Time and Date.
Thanks,
Cedric
Hi William,
can you clarify where the 20% stem from?
With the numbers you report (0.38s vs. 0.03s), it looks more like hyperapi is 10x slower.
However, I ran your benchmark locally with g++, and got exactly 0.0s for the "Custom function" due to constant folding.
clang++ on the other hand did actually run some code, and there I got numbers similar to yours (0.25s vs. 0.02s)
As a next step, I put the "getJulianDay" function into a separate compilation unit, after which it was significantly slower than hyperapi (0.25s vs. 0.40s)
I think the effects you are seeing here are mostly due to compiler microptimizations.
If you are interested in the topic, I can recommend this talk by Chandler Carruth: CppCon 2015: Chandler Carruth "Tuning C++: Benchmarks, and CPUs, and Compilers! Oh My!" - YouTube
Coming back to the issue at hand, Hyper uses the algorithm from the Calender FAQ (Section 2.15.1) to convert a Gregorian date into a Julian day number.
I put both yours and the hyperapi code into godbolt, and at least from a quick glance I cannot see much difference in the complexity of the generated assembly.
