Skip to main content

How to read hadoop ORC files from Vertica - need to pass hadoop/so username

  • July 17, 2017
  • 3 replies
  • 21 views

mmanzano

Hey there,
I am trying to read ORC files from a hadoop cluster with no success, I believe I have to pass (somehow) an username (they are validating user by O.S. user so, how to create an external table and in the COPY statement pass the username to be validated inside Hadoop cluster?!
Any clues?

I am using Vertica 8.1.1. over Red Hat 7.3
Regards,

3 replies

MaciejPaliwoda
Forum|alt.badge.img+2
  • Participating Frequently
  • July 17, 2017

We asked Customer - to copy them as ORC file onto regular filesystem.
Directory in our case had a special strcture:
DIR#level0: {Table Name}
DIR#level -1: {date ie: dt=2017-01-02}
DIR#level -2: {datafiles:ie: 000001_0, 00002_0, .....}

Then USED regular copy COPY {Table Name} FROM {datafiles} ORC ;

Important:
Check if Your customer does not use complex maptypes in ORC. Vertica does not support it.
ORC file can be also pointed as external table but we did not trying to get it from HDFS directly.

If you would like to import them one by one it is doable but remembert to point cluster node directly in COPY command (each file in a loop into different node). We had to do write a script to do it (let me know if you need it).

M.


mmanzano
  • Author
  • New Participant
  • July 18, 2017

Hi Maciej, how are you my friend?
The thing is: Hadoop personnel told me that they've implemented security validation via PAM (pam.d - you supply username/password and it returns what "thing" you can do), They are using that to control who will have access to the ORC files.

  They've told me that Hadoop cluster name is: orizoncluster so, I tried:

  COPY testtab FROM 'hdfs://orizoncluster/apps/hive/warehouse/dw_orizon.db/tb_layout_completo_orc/ano_mes=201701' ORC;

 But when I take a look at the LOAD_STREAMS, I got this error:

SELECT event_category, event_type, event_description, operator_name,
event_details, count(event_type) AS count
FROM query_events
WHERE event_type ilike 'LibHDFS++ FAILOVER RETRY'
GROUP BY event_category, event_type, event_description, operator_name, event_details;

-[ RECORD 1 ]-----+------------------------------------------
event_category | EXECUTION
event_type | LibHDFS++ FAILOVER RETRY
event_description | LibHDFS++ Namenode failover and retry.
operator_name | LibHDFS++ FileSystem
event_details | Libhdfs++ request failed on orizoncluster
count | 14

Then, I've stoped/cancelled the COPY command and then I got this error:

^CCancel request sent
WARNING 2005:
Could not connect to HDFS cluster (orizoncluster): Operation canceled
Could not accept config files in [/opt/hadoop/hcat/conf] because we were searching for hostname orizoncluster and the config files were for hostname altea01.orizon.local
Could not accept config files in [/opt/hadoop/hcat/conf] because we were searching for hostname orizoncluster and the config files were for hostname altea02.orizon.local
ERROR 7869: No files match when expanding glob: [hdfs://orizoncluster/apps/hive/warehouse/dw_orizon.db/tb_layout_completo_orc/ano_mes=201701]

Is there any configuration missing? What am I doing wrong!?

Thanks in advance!
Regards
MMAnzano


mmanzano
  • Author
  • New Participant
  • July 20, 2017

Hey guys, any clues?!
Regards,