Skip to main content
Question

Options/tools to migrate Vertica data to Azure Cloud ADLS

  • November 3, 2022
  • 6 replies
  • 15 views

vasanthvk

Hi,
I am looking for options/tools to migrate Vertica data to Azure cloud (ADLS), can someone please provide some guidance? Also, any reference architecture would be appreciated.
Thanks,
Vasanth Kumar

6 replies

SruthiA
Forum|alt.badge.img+1
  • Participating Frequently
  • November 3, 2022

You can use export to parquet and copy the data to azure

https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/SQLReferenceManual/Statements/EXPORTTOPARQUET.htm

Vertica can read external data created using ADLS Gen2, and data that Vertica exports can be read using ADLS Gen2.

https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/SQLReferenceManual/FileSystems/AzureBlob.htm


vasanthvk
  • Author
  • New Participant
  • November 3, 2022

Thank you, are there any other options the data size we are looking to migrate is from few GB's to 100 TB.


SruthiA
Forum|alt.badge.img+1
  • Participating Frequently
  • November 3, 2022

SruthiA
Forum|alt.badge.img+1
  • Participating Frequently
  • November 3, 2022

export to parquet is faster and parquet file size will also be way smaller compared to using other export options. It would be useful if you want to create external tables and query the parquet files present in ADLS


Bryan_H
Forum|alt.badge.img+2
  • Participating Frequently
  • November 3, 2022

If you partition your EXPORT, Vertica will parallelize the export based on number of partition keys and number of cores per node. So it's fairly fast and may only be limited to network bandwidth.

However, are you already on Azure? Or migrating to Azure? Is the intent to export data for other applications, or move the Vertica DB to the cloud? The Enterprise to Eon migration script can help move the entire DB to Eon mode on Azure.


mosheg
Forum|alt.badge.img+2
  • Participating Frequently
  • November 4, 2022

When you export to S3, GCS, or Azure Vertica writes files directly to the destination path, so you must wait for the export to finish before reading the files.