-
Story
-
Resolution: Done
-
Normal
-
None
-
RELENG Sprint 17
In OpenDaylight we have a script which takes care of cleaning up orphaned VMs.
https://github.com/opendaylight/releng-builder/blob/master/jjb/odl-openstack-cleanup-orphaned-nodes.sh
However there's a nasty race condition in this script where it's possible if the script happens to run at the same time Jenkins just spawned minions, it could issue a delete before the minion has attached to Jenkins.
How the script works is it fetches all of the active servers from openstack. Then fetches the list of active servers from Jenkins and compares them. Servers that are missing in Jenkins are thus deleted as the are likely orphaned. We should turn this into a function of lftools so that we can use Python to check the timestamp of when the server was created and if it's something less than 10 minutes ignore it as it's a freshly created server.