Skip to main content

Azure Data Lake Store with Vertica?

  • May 16, 2017
  • 2 replies
  • 11 views

peterjansens
Forum|alt.badge.img

We are talking to an MSP that is looking to build an azure based analytics platform (they are friendly to us and to MSFT),
The MSFT sales rep is competing by using FUD tactics, one of his remarks that I currently cannot answer is support for ‘Azure Data Lake Store’; it looks like an Hadoop dialect to me and includes Azure AD security, does that mean we can access it using external tables or maybe we should run a COPY against it to load the data?
Has anyone tried to use Azure Data Lake Store with Vertica?

2 replies

Car1os
Forum|alt.badge.img
  • Participating Frequently
  • June 13, 2017

I think it's Hadoop, based on the following:

HDFS for the cloud
Microsoft Azure Data Lake Store supports any application that uses the open Apache Hadoop Distributed File System (HDFS) standard. By supporting HDFS, you can easily migrate your existing Hadoop and Spark data to the cloud without recreating your HDFS directory structure.

So probably we can use our connectors to integrate with it.


peterjansens
Forum|alt.badge.img
  • Author
  • Participating Frequently
  • June 16, 2017

i agree, it is the Probably part that bugged me; i got a response from Chris Daly which i include here for reference;

Hi Peter,
While not an expert in the Azure Data Lake store, you are correct that it is based on Hadoop, Hortonworks to be exact. However I’m fairly sure that our HDFS connector will not work natively. Because the security for DLS is based on Active Directory, connections to the data in it seem to require a multistep authentication. This can be done externally to Vertica in Python, Node.js, or accessing the REST API which is probably the easiest. Since the process for reading data includes 3 steps – Authorization, and a two-step read command - creation of a UDx might be the best way to accomplish it.

Here is the link to the REST API commands for DLS
https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-rest-api