Skip to main content

PARTITION BY with Transform UDFs in R

  • February 8, 2014
  • 11 replies
  • 18 views
Said_Montiel
Forum|alt.badge.img+1
Hello,


     I have an R UDTF in vertica which takes a simple input and returns a matrix. I am trying to run this function with different sets of parameters (preferably in parallel) using the vertica cluster.

I am trying something like,

SELECT myFunc(col1 USING PARAMETERS x=1,y=2) OVER(PARTITION BY col1) FROM mainTable;


[Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokeProcessPartition(): Error calling processPartition() in User Defined Object [myFunc] at [/scratch_a/release/vbuild/vertica/UDxFence/RInterface.cpp:1236], error code: 0, message: Exception in processPartitionForR: [package ‘rJava’ could not be loaded] [SQL State=VP001, DB Errorcode=3399]

I am using rJava because within the UDTF I require several other pieces of information from queries to vertica.

Some help?

    11 replies

    Chris_C
    Forum|alt.badge.img
    • Chris_C
    • Participating Frequently
    • February 12, 2014
    Hi Said,

    Is the rJava package installed on Vertica's version of R?  The Vertica R binary is located in /opt/vertica/R/bin/R.  

    First make sure you have JDK 1.4+ installed on your system and the JAVA_HOME environment variable is set correctly.

    Then run the javareconf utility so R knows where Java is (you may have to be root or sudo to do this): 
    /opt/vertica/R/bin/R CMD javareconf
    Then run the Vertica packaged version of R and install the rJava package : 
    /opt/vertica/R/bin/R
    install.packages("rJava");
    quit();
    Then try to run your R UDF.

    Please let us know if this helps!

    Said_Montiel
    Forum|alt.badge.img+1
    • Said_Montiel
    • Author
    • Participating Frequently
    • February 19, 2014
    My Vertica cluster admin has run javareconf succesfully and he installed rJava as well. The rJava fails to load when using PARTITION BY as part of the UDF call inside the OVER() statement.

    If i try to execute without PARTITION BY it runs succesfully.

    I have removed this feature of the script and replaced its inner workings. Further research will be done and posted if new information arises.
    eli_revach
    Forum|alt.badge.img+2
    • eli_revach
    • Participating Frequently
    • July 27, 2014
    Do you able to find solution for this problem  ? , i am facing the same issue , R UDF run good without “partition by” and when use partition by  i keep seeing "[vertica-udx-R] <defunct>" process that being created  .

    PRana
    Forum|alt.badge.img
    • PRana
    • Participating Frequently
    • July 28, 2014
    Hi Eli,

    The [vertica-udx-R] <defunct> processes is a known issue, but it doesn't interfere with the query completion. Its just that the clean up once the query is done is not done properly. The defunct processes are cleaned up once the parent exits.

    Are your queries completing fine or do you get an error?

    Thanks
    Pratibha


    eli_revach
    Forum|alt.badge.img+2
    • eli_revach
    • Participating Frequently
    • July 29, 2014
    No error at the sql level , the udx log file include many . Please note that this issue is not related to specific r function , i get the same situation ehen i run the vertica sdk examples.
    PRana
    Forum|alt.badge.img
    • PRana
    • Participating Frequently
    • July 29, 2014
    Hi Eli,

    We are working on fixing the defunct processes issue. We do print some messages to the logs which are a part of the normal execution. As you know R is an interpreted process, we have to parse the R library look for certain optional functions like returntypescallback and if we don't find them we just print the fact to the logs.

    Pratibha
    eli_revach
    Forum|alt.badge.img+2
    • eli_revach
    • Participating Frequently
    • July 29, 2014
    For our implention is show stoper ,can you advice what is the eta for fix ?
    eli_revach
    Forum|alt.badge.img+2
    • eli_revach
    • Participating Frequently
    • July 29, 2014
    Can you send example of code that do not create this issue , it will good for us as a reference to see if something is wrongin in our code.
    eli_revach
    Forum|alt.badge.img+2
    • eli_revach
    • Participating Frequently
    • July 29, 2014
    We are 6.1.3 the workaround is not cosistent (not alwess shrink the defunk process) . i need hot fix asap . Iwill take it with suport. Thanks
    Said_Montiel
    Forum|alt.badge.img+1
    • Said_Montiel
    • Author
    • Participating Frequently
    • July 30, 2014
    Hi Eli!

    On this particular matter I can't remember what we did. But If you give me the last complete error line you encountered I can help you as well.

    On the side, we had to increase the JAVA memory limit value in order to give space to the query to execute. This helped us temporarily.

    =)
    eli_revach
    Forum|alt.badge.img+2
    • eli_revach
    • Participating Frequently
    • July 30, 2014
    Said ,
    Can you verify that you R function do not create [vertica-udx-R] <defunct> process on the vertica nodes ?
    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement