Steam reviews have always been kind of a double-edged sword. They’re a powerful way for customers to voice their dissatisfaction with a game, like in the case with Rockstar’s crackdown on Grand Theft Auto V mods getting reversed because of reviews. But it also means that developers can be review bombed by an extremely effective user base, which might have problems with a game beyond game design, such as the recent case with Taiwanese horror game Devotion. Developers have long complained to Valve about the latter problem, which Epic took into account when not having user reviews on their store. That decision may have prompted Valve into today’s announcement of overhauling user reviews on Steam.
In an extremely un-Valvelike move, the company has announced it is doing what it is often incredibly hesitant to do: hire a team of actual humans to do something. When a game gets an anomalous number of reviews at once, Valve dispatches a team of people to investigate the reviews and categorize non-relevant ones as off-topic. Review score went down because the game is broken? That’s relevant. Score went down because the reviewer said something people don’t like on Twitter? Likely off-topic. The thing is, what is determined to be “off-topic” seems broad enough to drive a truck through.
“We define an off-topic review bomb as one where the focus of those reviews is on a topic that we consider unrelated to the likelihood that future purchasers will be happy if they buy the game,” Valve says.
Valve clarified that some examples they consider when deciding is something is off-topic is complaints about a game’s end-user license agreement or digital rights management. So complaints about DRM like Denuvo slowing down Devil May Cry 5’s framerate would, presumably under Valve’s example, not contribute to its user score.
To be clear, this system does not filter bad reviews one-by-one. The team decides if a game is being review bombed and then declares all reviews within that period of time to be off-topic. That means good and bad reviews, so some might just have the bad luck of writing their praising review during a time where a flood of bad reviews that are irrelevant to the game’s quality came in. But at least humans are deciding it, so there’s more to it than Valve’s usual algorithm-based decision-making.
It remains to be seen how well this new plan will work or how the topics get defined as time goes on.
