On Saturday Feb 3rd, 2018. With the help of jconway we discovered the file system on OpenDaylight's Nexus instance ran out of inodes. We decided to mitigate this by clearing up some files to free up more inodes and returned Nexus back to normal operation.
On Monday Feb 5th, 2018. We discovered that the mitigation was not sufficient and ran into the inodes issue again. We decided to delete all jenkins092/builder-* jobs as they are very old and no longer necessary as well as jenkins092/docs-* jobs. This operation cleared up about 3 million inodes which hopefully will hold us off some more time. Will continue to monitor the system to ensure we are not rapidly running low on inodes.
The longer term solution is to migrate the file system to a new XFS file system. An rsync is in progress to move the files. Furthermore we decided in the future to put logs storage on a separate filesystem to isolate inodes issue if it happens again such that it doesn't cause a total outage of Nexus.
Proposed some documentation here https://gerrit.linuxfoundation.org/infra/8676