Implementing objective and subjective contract ratings

Burton took great pride in his ability to get his customer’s cargo from point A to point B quickly, intact, and with not a lot of questions asked. He’d invested tens of thousands of credits in tech to mask the signature of whatever he was carrying from UEE or militia scans. Burton considered himself among the best of the best in the hauler business, and his close friends and frequent customers agreed.

Burton’s problem? Not to put too fine a point on it, but ask around and everyone will tell you Burton’s a son-of-a-bitch. “I have no patience for stupid customers,” he’d growl at any customer – current or potential – who he thought was wasting his time. Burton’s favorite line? “Time is money, and if you want to keep wasting mine pull out your MobiGlas and start sending me credits.” Burton’s good friends, all two of them, told him he should scribe that line on the side of his ships.

When the Shippers’ Guild instituted customer satisfaction ratings in the 2930s, Burton’s business dropped like a stone. People liked his work, but always rated him down because of his abrasive personality. Burton was good enough he kept the business he stumbled across, but capturing new customers was always the toughest haul.

I hate to start any blog post with a reading assignment, but I have to with this one so we’re all on the same page when I say “contract”. You need to read my earlier blog post, “A Star Citizen Contract System.” Without understanding how comprehensive I believe the contract system inside Star Citizen should be, the rest of this might not make much sense.

The idea of rating the performance of contract counterparties has been around since the original pledge drive. In the initial pledge drive livestream, Chris Roberts talked about a Yelp-like system allowing the contract counterparties to rate each other subjectively. (FYI, the legal term for the two people who sign a contract is “counterparties”.) This cumulative subjective rating would be visible to other players as a guide for choosing good counterparties.

During today’s Wingman’s Hangar, CR talked about the ride-sharing service UBERx and how a similar rating system would be applied to contract performance in the Star Citizen universe. Unlike his Yelp comparison, this really got people excited on the forums – if only because between October 2012 and July 2013 we have several tens of thousands more backers to work each other into a frenzy.

I have to agree with the concerns of many: A 100% subjective rating system is prone to meaningful abuse. Gankers can get their friends to upvote them so they can use that good status to prey on other players. Worse, some jackass (as Burton would put it) would downvote you despite perfect contract performance just because he or she thought it would be funny.

If you assume CIG will work very hard to put in a comprehensive and robust contracting system, there is an easy way to address this.

Both players who sign a contract earn two ratings for that contract – a Yelp/UBERx subjective rating given by the contract counterparty AND a 100% objective rating calculated off the terms included in any contract.

Here’s how it would work for our friend Burton:

Burton signs a contract with Gracie. The contract says Burton has to ship 5,000 ton of artifacts between the Hades and Nul systems within 3 SEDs (Standard Earth Days, the time measurement system in the Star Citizen universe). The contract is worth 1,500 credits.

Burton picks up the artifacts in Hades, dodges some pirates, and delivers the goods as agreed to Gracie in the Nul system 2.5 SEDs later. Gracie thinks Burton is a little cute, so she starts making some small talk. Burton gives her his “Time is money” line, pissing Gracie off.

At the conclusion of the contract, Gracie gives Burton 3 out of 5 for performance because Burton hurt her feelings. That becomes Burton’s subjective rating for this contract. The CIG contracting system assigns a 5 out of 5 objective rating for delivering the goods intact and on time.

Burton gives Gracie 2 out of 5 for wasting his time, which is Gracie’s subjective rating on this contract. The CIG contracting system assigns a 5 out of 5 objective rating because Gracie paid Burton on time and the amount owed.

If I’m thinking about hiring Burton, and I don’t know him, I can look up his performance ratings. I will see a subjective rating of 3 and an objective rating of 5. I can make up my own mind which of the two ratings is more important to me. The same is true with Gracie’s subjective 2 and objective 5.

Having two ratings helps offset the ganking and jackassery inherent in a subjective rating system. If I run across someone like Burton with a near-perfect objective rating and a poor subjective rating, I might ask some questions about why the two ratings diverge so much. But having both of them available should provide the experienced Star Citizen player a nice picture of potential contract counterparties.

I’ll note the objective contract rating system could as easily be a contract completion percentage as opposed to a 1-5 scale rating. I’d argue for something sophisticated enough to provide small adjustments to the rating according to difficulty of the contract, but that’s not 100% necessary. Perhaps CIG might consider something patterned off how competitive diving is scored, with the performance of the contract modified up or down by a degree of difficulty measure.

I would also make individual ratings very accessible – showing not only the cumulative subjective and objective scores, but each contract rating and perhaps even some comments from the contract counterparty. This would provide even more visibility into your potential contract counterparty before “inking” the final contract.

I don’t expect CIG to get either the contract system or the rating system “right” on the first try, despite the brilliance of their team. The contract system, as you read in my blog post from February, is mind-numbingly complex. The rating system will have all sorts of little surprises in it, not the least of which is finding the right scale (0-5? 0-100?) and fine-tuning any adds and deducts to the objective rating.

I simply don’t think 100% reliance on a subjective rating system is workable. It’s just too gameable. The contracting system will (hopefully) be there, so why not make use of it?


2 thoughts on “Implementing objective and subjective contract ratings

  1. Another great post Eido!

    I personally prefer automatic rating systems based on given criteria with optional text feedback from players. The criteria would be a set of pre-defined objectives with the contractee determining the rewards for each. The contractor would be rated based on their accomplishments. The contractee would always get 5/5 since they are the party that pays up the dough.

    For missions where the contractor has to escort the contractee, the contractor won’t get any rewards if either the contractee perishes or the contractor strafes too far away from the contractee (ie abandons the mission). In all cases the contract would be terminated if the contractor perishes.

  2. I’m not sure how valuable the “subjective” ratings are. Regardless of the subjective rating, you can count on the result if you deal with someone with a high objective rating. If you have ever been in business for yourself, you fantasize about such a rating system. You would quickly decide that the subjective rating is useless, because only the objective relates to your bottom line. If you are wealthy enough and successful enough, you can afford to show attitude and pay attention to the subjective rating. But, then, YOUR subjective rating will probably be rather low.

Leave a Reply