Uploaded image for project: 'IT: Release Engineering'
  1. IT: Release Engineering
  2. RELENG-1215

Investigate odl nexus connectivity issues

Issue XMLXMLWordPrintable

    • RELENG Sprint 32, RELENG Sprint 33, RELENG Sprint 34

      From comments, outstanding issues:

      • 504 Gateway Timeouts with lftools deploy archives / Nexus unpack plugin
      • Connection timeouts caused by sockets in time_wait state, because incoming requests are exceeding 20-30k per/second on nginx. 

      What is addressed until now?

      • ODL Nexus is moved to larger flavor (8 cpu + 32 GiB).
      • ingress/router network latency issues are resolved.
      • Nexus schedule task (to rebuild maven metadata) is disabled.
      • Nexus schedule task (to delete old snapshots) is moved to Sundays.
      • Nexus cron jobs (purge old archives) was taking about 6-8hours to complete is fixed (CR in review).
      • rdiff backup script has been disabled. 
      • Identified socket/port exhaustion issue, expanded port range.
      • Identified an MTU issue between ingress and Nexus nodes, and that has been addressed.
      • FS tweaks and noatime is been put in place for ODL Nexus.
      • Network tunables in place for handling max connections, to address the time_wait on open sockets.  

      AI:

      • Pro-active monitoring and alearts are required Nexus system (and other internal nodes).     

      = = = = = = = = = = ====
      as PER

      https://rt-sso.linuxfoundation.org/Ticket/Display.html?id=60166

      this error from build logs

       02:34:02  [ERROR] Failed to execute goal on project daexim-impl: Could not resolve dependencies for project org.opendaylight.daexim:daexim-impl:bundle:1.3.4: Failed to collect dependencies at com.jayway.jsonpath:json-path-assert:jar:2.2.0: Failed to read artifact descriptor for com.jayway.jsonpath:json-path-assert:jar:2.2.0: Could not transfer artifact com.jayway.jsonpath:json-path-assert:pom:2.2.0 from/to opendaylight-mirror (https://nexus.opendaylight.org/content/repositories/public/): Network is unreachable (connect failed) -> [Help 1]

      might coorelate with this from odl-nexus

       [root@vex-yul-odl-nexus-1.ci logs]# grep 02:34 wrapper.log
jvm 1    | 2018-08-27 02:34:11 WARN  [32473851-542750] - org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Connection reset by peer [client=221.215.106.202,ua=Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E),req=GET http://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/karaf/0.8.3/karaf-0.8.3.zip]
jvm 1    | 2018-08-27 02:34:11 ERROR [32473851-542750] - org.sonatype.nexus.web.internal.ErrorPageFilter - Internal error
jvm 1    | 2018-08-27 02:34:13 WARN  [32473851-542742] - org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Connection reset by peer [client=221.215.106.202,ua=Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E),req=GET http://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/karaf/0.8.3/karaf-0.8.3.zip]
jvm 1    | 2018-08-27 02:34:13 ERROR [32473851-542742] - org.sonatype.nexus.web.internal.ErrorPageFilter - Internal error
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=central-new), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=maven-restlet), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=servicemix), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=juniper-contrail), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=sevntu), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=gemini-blueprint), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:14 WARN  [32473851-542732] - org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Remote HTTP request with malformed path attempted: repository M2Repository(id=clojars), path /com/huawei/bsp/com.huawei.bsp.commonlib.roa.restserver/${commonlib.version}/com.huawei.bsp.commonlib.roa.restserver-${commonlib.version}.jar
jvm 1    | 2018-08-27 02:34:51 INFO  [32473851-542713] - org.apache.http.impl.execchain.RetryExec - I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://svn.apache.org:80: The target server failed to respond
jvm 1    | 2018-08-27 02:34:51 INFO  [32473851-542713] - org.apache.http.impl.execchain.RetryExec - Retrying request to {}->http://svn.apache.org:80
jvm 1    | 2018-08-27 02:34:55 WARN  [32473851-542623] - org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Connection reset by peer [client=221.215.106.202,ua=Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E),req=GET http://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/karaf/0.8.3/karaf-0.8.3.zip]
jvm 1    | 2018-08-27 02:34:55 ERROR [32473851-542623] - org.sonatype.nexus.web.internal.ErrorPageFilter - Internal error
jvm 1    | 2018-08-27 11:02:34 WARN  [etcherImpl-task] - com.sonatype.central.secure.nexus.plugin.internal.AuthtokenFetcherImpl - Failed to fetch authtoken: org.apache.http.conn.ConnectTimeoutException: Connect to secure.central.sonatype.com:443 [secure.central.sonatype.com/207.223.241.90] failed: connect timed out
jvm 1    | 2018-08-27 13:02:34 INFO  [32473851-569336] - org.apache.http.impl.execchain.RetryExec - I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://svn.apache.org:80: The target server failed to respond
jvm 1    | 2018-08-27 13:02:34 INFO  [32473851-569336] - org.apache.http.impl.execchain.RetryExec - Retrying request to {}->http://svn.apache.org:80

      investigate further

        1. archives.zip
          8.47 MB
        2. cpu_sar_26.png
          cpu_sar_26.png
          154 kB
        3. cpu_steal_time_sar_27.png
          cpu_steal_time_sar_27.png
          109 kB
        4. find-cmds.png
          find-cmds.png
          48 kB
        5. find-cmds.png
          find-cmds.png
          48 kB
        6. find-cmds.png
          find-cmds.png
          48 kB
        7. image.png
          image.png
          27 kB
        8. image-2018-11-19-18-56-59-622.png
          image-2018-11-19-18-56-59-622.png
          116 kB
        9. image-2018-11-19-18-56-59-713.png
          image-2018-11-19-18-56-59-713.png
          189 kB
        10. image-2018-11-19-18-56-59-793.png
          image-2018-11-19-18-56-59-793.png
          177 kB
        11. image-2018-11-19-18-56-59-870.png
          image-2018-11-19-18-56-59-870.png
          192 kB
        12. image-2018-11-19-18-56-59-929.png
          image-2018-11-19-18-56-59-929.png
          155 kB
        13. image-2018-11-19-18-57-00-025.png
          image-2018-11-19-18-57-00-025.png
          129 kB
        14. image-2018-11-19-18-57-00-107.png
          image-2018-11-19-18-57-00-107.png
          186 kB
        15. image-2018-11-19-19-45-42-306.png
          image-2018-11-19-19-45-42-306.png
          116 kB
        16. image-2018-11-19-19-45-42-365.png
          image-2018-11-19-19-45-42-365.png
          177 kB
        17. image-2018-11-19-19-45-42-439.png
          image-2018-11-19-19-45-42-439.png
          192 kB
        18. image-2018-11-19-19-45-42-549.png
          image-2018-11-19-19-45-42-549.png
          189 kB
        19. image-2018-11-19-19-45-42-623.png
          image-2018-11-19-19-45-42-623.png
          155 kB
        20. image-2018-11-19-19-45-42-690.png
          image-2018-11-19-19-45-42-690.png
          186 kB
        21. image-2018-11-19-19-45-42-867.png
          image-2018-11-19-19-45-42-867.png
          129 kB
        22. image-2018-11-20-12-59-57-629.png
          image-2018-11-20-12-59-57-629.png
          157 kB
        23. image-2018-11-20-12-59-57-683.png
          image-2018-11-20-12-59-57-683.png
          203 kB
        24. image-2018-11-20-22-05-55-515.png
          image-2018-11-20-22-05-55-515.png
          109 kB
        25. io_sar_27.png
          io_sar_27.png
          145 kB
        26. load_sar_26.png
          load_sar_26.png
          162 kB
        27. net_sar_26.png
          net_sar_26.png
          154 kB
        28. net_sar_27.png
          net_sar_27.png
          123 kB
        29. sar-charts-16-17.tar.xz
          1.00 MB
        30. sar-charts-16-17.tar.xz
          1.00 MB
        31. Screenshot_20181115_174005.png
          Screenshot_20181115_174005.png
          299 kB

              askb Anil Belur
              jconway Jordan Conway
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day
                  1d