Uploaded image for project: 'IT: Release Engineering'
  1. IT: Release Engineering
  2. RELENG-2455

Rewrite pypi-release JJB templates to use test.pypi.org as staging repo

Issue XMLXMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Normal Normal
    • Q4 2019 Tech Debt
    • None
    • Global JJB
    • None

      Rewrite the PyPI release-verify and relase-merge JJB templates to pull binary artifacts from test.pypi.org (or other repo) and push to pypi.org (or other repo), based on PyPY project and release (distribution) strings.

      The first draft of PyPI support has release-verify and release-merge templates that accept a release YAML file as instructions, but build from source which differs sharply from the Maven and Docker release jobs.  The Python index server test.pypi.org is perfect for the task of staging repository, because it allows upload of a file at a given version exactly once, which is even more stringent than Nexus, basically treats everything as a release.

      The new release-verify job should do the following:

      1. Run pip download to fetch files from the test index server (default pypi-test, which means test.pypi.org), and if there's a match, continue
      2. Run pip download to fetch files from the production index server (default pypi, which means pypi.org), and if there's NO match, continue.
      3. Otherwise fail.

      The new release-merge job should do the following:

      1. Same as verify
      2. Same as verify
      3. Upload files to pypi.org

      The release yaml file schema requires a "project" key but it's not currently used. That key should carry the python project name (not the gerrit/git repo name), and the shell script(s) must query the PyPI project and the release/distribution strings from the release yaml file. This will allow support of multiple Python modules in a single Gerrit repository, which is common at ONAP.

      A significant wrinkle: pip download checks python version against the package requested, and will signal error if files exist but are not compatible. The release yaml file MAY need to specify the python version for supplying on the pip download line, that's kind of ugly, but the person requesting release should not have to worry about which Python3 is executing the release job. Is there a better way?  Pip doesn't seem to have a "ignore version compatibility" flag.

      Another significant issue: twine is directed to a PyPI index via the .pypirc dot-ini file which has a URL like https://test.pypi.org/legacy/.  But pip needs a "dash i" argument to specify which index to use, and that requires a URL like https://test.pypi.org/simple. In other words, they use different suffixes. This is a bit of a disconnect unfortunately. I guess upload script will need to extract the URL from the .pypirc, keep the base and append a suffix?  But that's quite a hack.

      Another wrinkle, pip download can drag along dependencies, don't forget the flag to exclude them.

              cl778h Chris Lott
              cl778h Chris Lott
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: