Skip to main content

Table creation/varchar/Dataiku

  • February 7, 2018
  • 4 replies
  • 13 views

Bernard92
Forum|alt.badge.img+2

Hi all,

One of our PS customer is using Dataiku with Vertica , and when Dataiku creates table with Varchar columns , they all default to Varchar(16200)
Any idea why , and if this will be a problem for Vertica ?
should we change this and how ?

Thanks
Bernard

4 replies

Ben_Vandiver
Forum|alt.badge.img
  • Participating Frequently
  • February 7, 2018

Not familiar with Dataiku, but declaring everything as a wide varchar can negatively impact query performance.


Bernard92
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • February 7, 2018

That's exactly why I am concerned about. However I'm still very unclear how Vertica behave with these wide varchar columns , could you maybe tell me a bit more about ?
The customer was also concerned about the potential impact on the raw database size and the licence


Francois
Forum|alt.badge.img+1
  • Participating Frequently
  • February 7, 2018

Hello,
from my experience with Dataiku, we do have a potential problem on performance with this
Varchar (16200) is the defaut data type used in generic table creation by dataiku
they also use "wide" numeric data type (seen recently numeric(38.10)
So you can get from a dataiku job a wide table creation and load (hundreds of columns) with wide data types (generic "string" always created as varchar(16200)
It has direct impact on load performance with slow process on insert...select
I have been working recently with a customer where a dataiku job of around 4 hour was reduced to 30 min when using "reasonnable" data types

Customer are supposed to act on this datatype choice from Dataiku
See "data type limitation" in
https://my.vertica.com/kb/HPE-Vertica-Integration-with-Dataiku-Connection-Guide/Content/Partner/HPE-Vertica-Integration-with-Dataiku-Connection-Guide.htm

By the way it should not have ay impact oon raw database size, we do not use tehe column definition


Bernard92
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • February 7, 2018

Thanks François , that's very very useful