I did something similar, in fact, the math may be the same thing and just expressed differently. But when I've had to rank non-transitive things, I use Elo (https://en.wikipedia.org/wiki/Elo_rating_system)
Many years ago when I was a mid-level developer at a dysfunctional company, I was senior enough to be invited to some "strategy" meetings, but junior enough that no one ever listened to me. We (engineering, sales, marketing, etc.) spent nearly an entire summer bickering over what "important" features we were going to schedule next. I finally got fed up, took everything out of the ticketing system and made random parings and had people vote on it. Then just like a chess match, updated their Elo score based on the outcome. Then I had anyone who cared play match-ups for as long as they wanted. We ended up getting a decent ordering of features and finally ended the summer of hell meetings.
I don't know if the order was the correct order, I didn't stay around long enough to see. I was just happy that sales and marketing folks thought that I had some magic math that solved their problem, and I was happy to be back developing and not sitting in useless meetings.
What I like about this is that you don't have to be self consistent, as long as on average you pick the best, it will bubble to the top. And you can mix the results of other voters and see what the "true" winner is. (Of course, to be fair, you have to give each person the same number of match ups, in my case, I just served match-ups to anyone who wanted to sit at the terminal and vote, so someone could have wasted an entire day and overwhelm the system - I didn't care at the time).
Many years ago when I was a mid-level developer at a dysfunctional company, I was senior enough to be invited to some "strategy" meetings, but junior enough that no one ever listened to me. We (engineering, sales, marketing, etc.) spent nearly an entire summer bickering over what "important" features we were going to schedule next. I finally got fed up, took everything out of the ticketing system and made random parings and had people vote on it. Then just like a chess match, updated their Elo score based on the outcome. Then I had anyone who cared play match-ups for as long as they wanted. We ended up getting a decent ordering of features and finally ended the summer of hell meetings.
I don't know if the order was the correct order, I didn't stay around long enough to see. I was just happy that sales and marketing folks thought that I had some magic math that solved their problem, and I was happy to be back developing and not sitting in useless meetings.
What I like about this is that you don't have to be self consistent, as long as on average you pick the best, it will bubble to the top. And you can mix the results of other voters and see what the "true" winner is. (Of course, to be fair, you have to give each person the same number of match ups, in my case, I just served match-ups to anyone who wanted to sit at the terminal and vote, so someone could have wasted an entire day and overwhelm the system - I didn't care at the time).