Uploaded image for project: 'IT: Release Engineering'
  1. IT: Release Engineering
  2. RELENG-2746

FD.io - Nexus decommission plan

    Issue XMLXMLWordPrintable

Details

    Description

      This ticket is to document community requirements and the migration plan to decommission Nexus

      Description from: IT-23352
      FD.io project data (logs, archived artifacts, & docs) are now being uploaded to Amazon S3 buckets instead of nexus.fd.io. In order to complete the retirement of nexus.fd.io, data currently residing on nexus shall be migrated to amazon S3.

      This ticket will be used to document what needs to be migrated from nexus, where it will be migrated to in amazon s3, and track the completion status of the migration.

      Attachments

        JEditor

          Activity

            Current Status:

            Done
            38 weeks, 6 days, 15 minutes
            valderrv Vanessa Valderrama transitioned issue -
            Transition Name Total Time in Status
            Create
            to Backlog
            valderrv Vanessa Valderrama created issue -
            valderrv Vanessa Valderrama transitioned issue -
            To Do
            Backlog to Selected for Development
            11 seconds
            valderrv Vanessa Valderrama made changes -
            Field Original Value New Value
            Status Backlog [ 10701 ] Selected for Development [ 10714 ]
            valderrv Vanessa Valderrama made changes -
            Epic Link RELENG-2643 [ 72127 ]
            cpanell Casey Panell made changes -
            Component/s Project: FD.io [ 12502 ]
            valderrv Vanessa Valderrama transitioned issue -
            Backlog
            Selected for Development to Backlog
            5 weeks, 5 days, 22 hours, 5 minutes
            valderrv Vanessa Valderrama made changes -
            Status Selected for Development [ 10714 ] Backlog [ 10701 ]
            cpanell Casey Panell made changes -
            Epic Link RELENG-2643 [ 72127 ] RELENG-2841 [ 78474 ]
            agrimberg Andrew Grimberg made changes -
            Issue Type Task [ 3 ] Story [ 10001 ]
            valderrv Vanessa Valderrama transitioned issue -
            To Do
            Backlog to Selected for Development
            31 weeks, 6 days, 6 hours, 16 minutes
            valderrv Vanessa Valderrama made changes -
            Status Backlog [ 10701 ] Selected for Development [ 10714 ]
            valderrv Vanessa Valderrama transitioned issue -
            In Progress
            Selected for Development to In Progress
            4 seconds
            valderrv Vanessa Valderrama made changes -
            Status Selected for Development [ 10714 ] In Progress [ 10201 ]

            Added to the TSC agenda to start discussion

            valderrv Vanessa Valderrama added a comment - Added to the TSC agenda to start discussion
            valderrv Vanessa Valderrama made changes -
            Epic Link RELENG-2841 [ 78474 ]

            Docs - where will we migration docs to
            Logs/Artifact archives - S3
            Document a list of projects still deploying artifacts to Nexus

            valderrv Vanessa Valderrama added a comment - Docs - where will we migration docs to Logs/Artifact archives - S3 Document a list of projects still deploying artifacts to Nexus

            Deploying to Nexus & PackageCloud

            • CICN
            • CSIT (docs)
            • HC2VPP
            • HICN
            valderrv Vanessa Valderrama added a comment - Deploying to Nexus & PackageCloud CICN CSIT (docs) HC2VPP HICN
            valderrv Vanessa Valderrama transitioned issue -
            Blocked
            In Progress to Blocked
            6 days, 25 seconds
            valderrv Vanessa Valderrama made changes -
            Status In Progress [ 10201 ] Blocked [ 10002 ]

            Pending further TSC discussion

            valderrv Vanessa Valderrama added a comment - Pending further TSC discussion
            anusinha Anubha Sinha made changes -
            Epic Link RELENG-3238 [ 96437 ]
            valderrv Vanessa Valderrama transitioned issue -
            Backlog
            Blocked to Backlog
            2 weeks, 2 days, 23 hours, 41 minutes
            valderrv Vanessa Valderrama made changes -
            Status Blocked [ 10002 ] Backlog [ 10701 ]
            valderrv Vanessa Valderrama transitioned issue -
            To Do
            Backlog to Selected for Development
            3 seconds
            valderrv Vanessa Valderrama made changes -
            Status Backlog [ 10701 ] Selected for Development [ 10714 ]
            valderrv Vanessa Valderrama transitioned issue -
            Backlog
            Selected for Development to Backlog
            4 days, 19 hours, 47 minutes
            valderrv Vanessa Valderrama made changes -
            Status Selected for Development [ 10714 ] Backlog [ 10701 ]
            vrpolak Vratko Polak added a comment - Comments regarding CSIT: https://lists.fd.io/g/csit-dev/message/4251

            VPP docs jobs also deploy to nexus, don’t they ?

            ayourtch Andrew Yourtchenko added a comment - VPP docs jobs also deploy to nexus, don’t they ?
            vrpolak Vratko Polak added a comment - Yes. https://docs.fd.io/vpp/

            See further comments regarding CSIT project: https://lists.fd.io/g/csit-dev/message/4252.

            mackonstan Maciek Konstantynowicz added a comment - See further comments regarding CSIT project: https://lists.fd.io/g/csit-dev/message/4252 .
            valderrv Vanessa Valderrama made changes -
            Epic Link RELENG-3238 [ 96437 ]
            anusinha Anubha Sinha made changes -
            Fix Version/s Q4 2020 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 15501 ]
            anusinha Anubha Sinha made changes -
            Epic Link RELENG-3238 [ 96437 ]
            anusinha Anubha Sinha made changes -
            Epic Link RELENG-3238 [ 96437 ] RELENG-3357 [ 110815 ]
            Fix Version/s Q4 2020 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 15501 ]
            valderrv Vanessa Valderrama made changes -
            Epic Link RELENG-3357 [ 110815 ]
            valderrv Vanessa Valderrama made changes -
            Summary FD.io - Nexus to PackageCloud migration FD.io - Nexus decommission plan
            valderrv Vanessa Valderrama transitioned issue -
            To Do
            Backlog to Selected for Development
            38 weeks, 1 day, 21 hours, 54 minutes
            valderrv Vanessa Valderrama made changes -
            Status Backlog [ 10701 ] Selected for Development [ 10714 ]

            Document migration plan

            • docs migration
            • artifacts migration
            • jjb changes
            valderrv Vanessa Valderrama added a comment - Document migration plan docs migration artifacts migration jjb changes
            anusinha Anubha Sinha made changes -
            Epic Link RELENG-3807 [ 138416 ]
            Fix Version/s Q3 2021 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 18005 ]
            anusinha Anubha Sinha made changes -
            Fix Version/s Q4 2021 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 18413 ]
            Fix Version/s Q3 2021 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 18005 ]
            anusinha Anubha Sinha made changes -
            Epic Link RELENG-3807 [ 138416 ] RELENG-4023 [ 152015 ]
            valderrv Vanessa Valderrama made changes -
            Description Need to reach out to any projects still using Nexus to see if they are willing to move from Nexus to PackageCloud

            TSC meeting notes
            http://ircbot.wl.linuxfoundation.org/meetings/fdio-meeting/2020/fdio-meeting.2020-02-20-16.00.log.html
            valderrv Vanessa Valderrama made changes -
            Description This ticket is to document community requirements and the migration plan to decommission Nexus
            valderrv Vanessa Valderrama made changes -
            Link This issue relates to IT-23352 [ IT-23352 ]
            valderrv Vanessa Valderrama made changes -
            Description This ticket is to document community requirements and the migration plan to decommission Nexus This ticket is to document community requirements and the migration plan to decommission Nexus

            Description from: IT-23352
            FD.io project data (logs, archived artifacts, & docs) are now being uploaded to Amazon S3 buckets instead of nexus.fd.io. In order to complete the retirement of nexus.fd.io, data currently residing on nexus shall be migrated to amazon S3.

            This ticket will be used to document what needs to be migrated from nexus, where it will be migrated to in amazon s3, and track the completion status of the migration.
            valderrv Vanessa Valderrama made changes -
            Watchers Andrew Yourtchenko, Maciek Konstantynowicz, Vanessa Valderrama, Vratko Polak [ Andrew Yourtchenko, Maciek Konstantynowicz, Vanessa Valderrama, Vratko Polak ] Andrew Yourtchenko, Dave Wallace, Ed Warnicke, Maciek Konstantynowicz, Peter Mikus, Vanessa Valderrama, Vratko Polak [ Andrew Yourtchenko, Dave Wallace, Ed Warnicke, Maciek Konstantynowicz, Peter Mikus, Vanessa Valderrama, Vratko Polak ]
            pmikus Peter Mikus added a comment -

            From CSIT point of view:

            https://docs.fd.io/csit/

            rls1701/ 2017-02-01 16:02 -  
            rls1704/ 2017-04-27 19:00 -  
            rls1707/ 2017-07-21 13:51 -  
            rls1710/ 2017-10-19 12:53 -  
            rls1801/ 2018-03-29 08:56 -  
            rls1801_2/ 2018-06-01 05:53 -  
            rls1804/ 2018-05-03 08:24 -  
            rls1807/ 2018-08-08 10:50 -  
            rls1810/ 2018-11-07 15:15 -  
            rls1901/ 2019-02-12 15:27 -  
            rls1901_1/ 2019-03-19 17:35 -  
            rls1901_2/ 2019-05-27 12:29 -  
            rls1901_3/ 2019-08-07 11:58 -  
            rls1904/ 2019-05-10 07:51 -  
            rls1904_1/ 2019-06-12 17:42 -  
            rls1904_2/ 2019-08-09 06:25 -  
            rls1908/ 2019-09-04 05:24 -  
            rls1908_1/ 2019-09-25 11:19 -  
            rls1908_2/ 2020-05-06 15:00 -  
            rls2001/ 2020-03-29 04:22 -  
            rls2005/ 2020-06-17 09:08 -  
            rls2009/ 2020-10-14 11:38 -  
            rls2009_lts/ 2020-12-02 07:17 -  
            rls2101/ 2021-02-10 08:57 -  
            rls2101_1/ 2021-08-18 11:35 -  
            rls2106/ 2021-07-14 15:34 -  

             

            Please DO NOT copy master folder/branch.

            Thank you.

            pmikus Peter Mikus added a comment - From CSIT point of view: https://docs.fd.io/csit/ rls1701/ 2017-02-01 16:02 -   rls1704/ 2017-04-27 19:00 -   rls1707/ 2017-07-21 13:51 -   rls1710/ 2017-10-19 12:53 -   rls1801/ 2018-03-29 08:56 -   rls1801_2/ 2018-06-01 05:53 -   rls1804/ 2018-05-03 08:24 -   rls1807/ 2018-08-08 10:50 -   rls1810/ 2018-11-07 15:15 -   rls1901/ 2019-02-12 15:27 -   rls1901_1/ 2019-03-19 17:35 -   rls1901_2/ 2019-05-27 12:29 -   rls1901_3/ 2019-08-07 11:58 -   rls1904/ 2019-05-10 07:51 -   rls1904_1/ 2019-06-12 17:42 -   rls1904_2/ 2019-08-09 06:25 -   rls1908/ 2019-09-04 05:24 -   rls1908_1/ 2019-09-25 11:19 -   rls1908_2/ 2020-05-06 15:00 -   rls2001/ 2020-03-29 04:22 -   rls2005/ 2020-06-17 09:08 -   rls2009/ 2020-10-14 11:38 -   rls2009_lts/ 2020-12-02 07:17 -   rls2101/ 2021-02-10 08:57 -   rls2101_1/ 2021-08-18 11:35 -   rls2106/ 2021-07-14 15:34 -     Please DO NOT copy master folder/branch. Thank you.
            agrimberg Andrew Grimberg made changes -
            Epic Link RELENG-4023 [ 152015 ] RELENG-4085 [ 164792 ]
            Fix Version/s Q1 2022 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 19600 ]
            Fix Version/s Q4 2021 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 18413 ]
            qpwoei Gau Pan made changes -
            Rank Ranked higher
            pmikus Peter Mikus added a comment - - edited

            Replication of CSIT data .... [done]  (both report/doc)

            e.g: https://s3-docs.fd.io/csit/rls1701/report/

            Replication of VPP to be confirmed (based on requirement)

            pmikus Peter Mikus added a comment - - edited Replication of CSIT data .... [done]   (both report/doc) e.g:  https://s3-docs.fd.io/csit/rls1701/report/ Replication of VPP to be confirmed (based on requirement)
            hagbard Ed Warnicke added a comment -

            @Peter, it looks like you are setting up for s3-docs.fd.io to become docs.fd.io once we are comfortable that we are prepared so that we get a smooth transition, is that correct?

            hagbard Ed Warnicke added a comment - @Peter, it looks like you are setting up for s3-docs.fd.io to become docs.fd.io once we are comfortable that we are prepared so that we get a smooth transition, is that correct?
            pmikus Peter Mikus added a comment -

            @Ed, I would say, we did transition, in all our scripts and ci-man, backlinks we are using S3. The only missing part was old data (21.06 and earlier). To whatever degree any project wants to transfer them. 

            from CSIT point of view, all done. (140G data)

            from VPP point of view, script for migration of "/docs" is now running. (it is very slow as the old data between16.09..21.06 releases counts 1,368,068 files // 32G)

            from <any_other_project> point ov view... ????? (e.g back up Honeycomb makes no sense as it is closed). So I suggest to ask via TSC "which projects want to have docs.fd.io transferred? #poll"

            Earlier we agreed that transferring jenkins logs is worthless, we are only talking about docs.

             

             

            Overall I will let you know once VPP is done.

            pmikus Peter Mikus added a comment - @Ed, I would say, we did transition, in all our scripts and ci-man, backlinks we are using S3. The only missing part was old data (21.06 and earlier). To whatever degree any project wants to transfer them.  from CSIT point of view, all done. (140G data) from VPP point of view, script for migration of "/docs" is now running. (it is very slow as the old data between16.09..21.06 releases counts 1,368,068 files // 32G) from <any_other_project> point ov view... ????? (e.g back up Honeycomb makes no sense as it is closed). So I suggest to ask via TSC "which projects want to have docs.fd.io transferred? #poll" Earlier we agreed that transferring jenkins logs is worthless, we are only talking about docs.     Overall I will let you know once VPP is done.
            hagbard Ed Warnicke added a comment -

            What's the plan to make sure that a doc.fd.io link continues to work after we decommission nexus?  Will we simply make doc.fd.io a CNAME for s3-docs.fd.io ?

             

            What is the plan for indexes (the equivalent of this link https://docs.fd.io/vpp/ ) to make sure folks can find stuff? 

            hagbard Ed Warnicke added a comment - What's the plan to make sure that a doc.fd.io link continues to work after we decommission nexus?  Will we simply make doc.fd.io a CNAME for s3-docs.fd.io ?   What is the plan for indexes (the equivalent of this link https://docs.fd.io/vpp/  ) to make sure folks can find stuff? 
            valderrv Vanessa Valderrama transitioned issue -
            In Progress
            Selected for Development to In Progress
            22 weeks, 4 days, 4 hours, 46 minutes
            valderrv Vanessa Valderrama made changes -
            Status Selected for Development [ 10714 ] In Progress [ 10201 ]

            hagbard
            Are you asking about an index doc? Please look over this Amazon documentation and let me know if this is what you're referring to
            https://docs.aws.amazon.com/AmazonS3/latest/userguide/IndexDocumentSupport.html

            valderrv Vanessa Valderrama added a comment - hagbard Are you asking about an index doc? Please look over this Amazon documentation and let me know if this is what you're referring to https://docs.aws.amazon.com/AmazonS3/latest/userguide/IndexDocumentSupport.html
            hagbard Ed Warnicke added a comment -

            That works to a point, except that it appears we would then need to manually create any relevant index document.  That might work, but we will need to take some care to get it right, including needing to update those index documents as new versions of things (or new elements of directory structure) land.

             

             

            hagbard Ed Warnicke added a comment - That works to a point, except that it appears we would then need to manually create any relevant index document.  That might work, but we will need to take some care to get it right, including needing to update those index documents as new versions of things (or new elements of directory structure) land.    
            pmikus Peter Mikus added a comment -

            Yes to manually create (in fact it already exists) I just need to upload it.
            NO!!! to change indexing settings for docc bucket as it would break those docs sites.

             

            pmikus Peter Mikus added a comment - Yes to manually create (in fact it already exists) I just need to upload it. NO!!! to change indexing settings for docc bucket as it would break those docs sites.  
            pmikus Peter Mikus added a comment -

            CNAME for docs.fd.io is fine. but please do not remove s3-docs.fd.io as it is now hardcoded in many places.

            pmikus Peter Mikus added a comment - CNAME for docs.fd.io is fine. but please do not remove s3-docs.fd.io as it is now hardcoded in many places.
            hagbard Ed Warnicke added a comment -

            Peter: Why is it hard coded?

            hagbard Ed Warnicke added a comment - Peter: Why is it hard coded?
            pmikus Peter Mikus added a comment -

            @Ed: How would you today address s3 only storage? I am not a fan of waiting till someone flip a switch. We are using s3-docs for over a 6mnths. Nexus should be down already. So... 

            pmikus Peter Mikus added a comment - @Ed: How would you today address s3 only storage? I am not a fan of waiting till someone flip a switch. We are using s3-docs for over a 6mnths. Nexus should be down already. So... 
            hagbard Ed Warnicke added a comment -

            @Peter: Let me ask a different question that probably will help the discussion:

            Where are we referring to s3-docs.fd.io ?

            hagbard Ed Warnicke added a comment - @Peter: Let me ask a different question that probably will help the discussion: Where are we referring to s3-docs.fd.io ?
            pmikus Peter Mikus added a comment - https://github.com/FDio/site/search?q=s3-docs https://github.com/FDio/csit/search?q=s3-docs https://github.com/FDio/ci-management/search?q=s3-docs  

            Discussed during TSC today that we will pick up the decom discussion after the release

            valderrv Vanessa Valderrama added a comment - Discussed during TSC today that we will pick up the decom discussion after the release
            valderrv Vanessa Valderrama transitioned issue -
            Blocked
            In Progress to Blocked
            2 weeks, 22 hours, 26 minutes
            valderrv Vanessa Valderrama made changes -
            Status In Progress [ 10201 ] Blocked [ 10002 ]

            pmikus hagbard dwallacelf are we ready to start discussing the decom again?

            I think the next steps would be to figure out what to do with the index page and finalize what needs to migrate.

            valderrv Vanessa Valderrama added a comment - pmikus hagbard dwallacelf are we ready to start discussing the decom again? I think the next steps would be to figure out what to do with the index page and finalize what needs to migrate.
            anusinha Anubha Sinha made changes -
            Epic Link RELENG-4085 [ 164792 ] RELENG-4197 [ 178266 ]
            Fix Version/s Q2 2022 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 20415 ]
            Fix Version/s Q1 2022 RelEng Operations (Enhancements, Tech Debt, Build, Release Support) [ 19600 ]

            Per discussion with Dave, all artifacts have been migrated. Nexus is ready for decom.

            valderrv Vanessa Valderrama added a comment - Per discussion with Dave, all artifacts have been migrated. Nexus is ready for decom.
            valderrv Vanessa Valderrama transitioned issue -
            Done
            Blocked to Done
            9 weeks, 5 days, 21 hours, 42 minutes
            valderrv Vanessa Valderrama made changes -
            Resolution Done [ 10000 ]
            Status Blocked [ 10002 ] Done [ 10200 ]

            People

              valderrv Vanessa Valderrama
              valderrv Vanessa Valderrama
              Andrew Yourtchenko, Dave Wallace, Ed Warnicke, Maciek Konstantynowicz, Peter Mikus, Vanessa Valderrama, Vratko Polak
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Salesforce