Skip to main content
Question

Are there some configuration parameters controlling the throughput of HDFS storage location?

  • March 30, 2018
  • 4 replies
  • 13 views

Dingqiang
Forum|alt.badge.img+1

Hi team,
Sorry bother you with this question as caption. I have a storage location on HDFS, but no more than 50MB throughput even there are multiple queries or batch INSERT /*+ direct */ or COPY statements. HDFS should not be the bottleneck as Hive can give much higher throughput to HDFS/network.
My Vertica is v9.0.1-1.
Any suggestion?

Thanks & regards,
DQ

4 replies

Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • March 30, 2018

Did you measure the storage locations throughput with the MEASURE_LOCATION_PERFORMANCE function?

See:
https://my.vertica.com/docs/9.0.x/HTML/index.htm#Authoring/SQLReferenceManual/Functions/VerticaFunctions/MEASURE_LOCATION_PERFORMANCE.htm

There are a bunch of database parameters for Hadoop:

select parameter_name, description from vs_configuration_parameters where parameter_name ilike '%hdfs%' or parameter_name ilike '%hadoop%';

Do you have Vertica installed on the same cluster as Hadoop? If not, maybe you have a network bottleneck between Vertica and Hadoop?


Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • March 30, 2018

The parameter HadoopFSReplication controls the number of copies of data HDFS stores. Maybe reducing it to 1 will help you?

See:
https://my.vertica.com/docs/9.0.x/HTML/index.htm#Authoring/HadoopIntegrationGuide/StorageLocations/TroubleSHootingHDFSStorageLocations.htm


Dingqiang
Forum|alt.badge.img+1
  • Author
  • Participating Frequently
  • April 3, 2018

Thank you Jim. But it seems HadoopFSReplication came from webhdfs filesystem era, does it work for our NativeHadoopFileSystem?


Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • April 3, 2018

I assume the example in the doc here:

https://my.vertica.com/docs/9.0.x/HTML/index.htm#Authoring/HadoopIntegrationGuide/StorageLocations/TroubleSHootingHDFSStorageLocations.htm

Is using the a storage location defined using the hdfs:/// URL.

I just sent an email asking the doc team to add a CREATE LOCATION to the example.

=> CREATE LOCATION 'hdfs:///user/dbadmin' ALL NODES SHARED USAGE 'data' LABEL 'hdfs';

I think that would be more clear :smile: as the earlier examples create a storage location labeled "coldstorage".