Skip to main content
Question

Seeing Broken pipe errors after upgrading the Vertica nodes from Linux 6 to 7.

  • September 7, 2022
  • 17 replies
  • 10 views
gjabraj
Forum|alt.badge.img

`Caused by: java.sql.SQLRecoverableException: [Vertica]VJDBC IOException while communicating with server: java.net.SocketException: Broken pipe (Write failed).
at com.vertica.io.ProtocolStream.logAndConvertToNetworkException(Unknown Source)
at com.vertica.io.ProtocolStream.flush(Unknown Source)
at com.vertica.core.VConnection.handleError(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeUpdate(Unknown Source)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTable(VerticaTableSQL.java:430)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTable(VerticaTableSQL.java:379)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTables(VerticaTableSQL.java:328)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.finalize(CrunchJob.java:436)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.execute(CrunchJob.java:390)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.executeJob(CrunchJob.java:220)
at com.xx.management.vertica.job.ESMJobManager.executeJob(ESMJobManager.java:88)
... 9 more
Caused by: com.vertica.support.exceptions.RecoverableException: [Vertica]VJDBC IOException while communicating with server: java.net.SocketException: Broken pipe (Write failed).
... 24 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.vertica.io.VStream.flush(Unknown Source)
... 23 more

Build step 'Execute shell' marked build as failure
Finished: FAILURE`

    17 replies

    Bryan_H
    Forum|alt.badge.img+2
    • Bryan_H
    • Participating Frequently
    • September 7, 2022

    What specifically causes the exception? That is, was it on connect, or a query, or something else like a prepared statement?
    Do other clients (JDBC, vsql, etc.) still work from the same environment?
    Was the JDK upgraded also during the OS upgrade? Did any OS settings change, such as firewall or iptables?

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • September 8, 2022

    We have hadoop jobs Inserting data to Vertica. If the job runs over 1 hour or more it fails with above error.

    No JDK is installed on the Vertica nodes.

    No firewalls/ACL's present on the Vertica cluster.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • September 8, 2022

    is it copy from HDFS which is failing?

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • September 9, 2022

    @SruthiA said:
    is it copy from HDFS which is failing?

    Yes. HDFS data is inserted to Vertica.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • September 12, 2022

    how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • September 14, 2022

    @SruthiA said:
    how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?

    JDBC being used is vertica-jdbc-8.1.1-18.jar.

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • September 15, 2022

    @SruthiA : Will upgrading the vertica-jdbc to higher version help??

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • September 15, 2022

    @gjabraj issue looks like connection time out based on the errors. please run one long running COPY statement via vsql to see if completes successfully. it will confirm that issue is with long running jobs via JDBC. do you see any errors in vertica log when it fails with error?

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • September 21, 2022

    @SruthiA : Looks like vsql does not have an issue being idle over an hour. Our next plan is to upgrade vertica-jdbc jar to latest. Please let us know otherwise.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • September 21, 2022

    @gjabraj Thank you for confirming. Please try using the latest JDBC driver and see if it helps. if there is any load balancer between vertica and your client using JDBC driver try check if that is disconnecting the idle sessions

    KennethAman
    • KennethAman
    • New Participant
    • October 3, 2022

    @SruthiA said:
    how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?

    JDBC being used is vertica-jdbc-8.1.1-18.jar

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • October 11, 2022

    @SruthiA : After updating the jdbc driver to latest version we are still seeing the same issue.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • October 11, 2022

    @gjabraj : It looks like we need to review it on the webEx. Could you please open a support case? what is the vertica version.

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • October 11, 2022

    Current Vertica version is 10.1.x. Upgrade to 11.x scheduled next week.

    Support case will be logged soon.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • October 11, 2022

    When you upgrade to 11.x and if you have changed keepalive settings at OS level, make sure you change it in vertica DB.. we introduced new config parameters for keepalive settings in 11.x

    https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/AdministratorsGuide/ManagingClientConnections/OverviewClientConnections.htm

    gjabraj
    Forum|alt.badge.img
    • gjabraj
    • Author
    • Participating Frequently
    • October 20, 2022

    @SruthiA : Logged this case 02428942

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • October 21, 2022

    @gjabraj : Thank you. Assigned engineer will get back to you soon.

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement