Amazon is failing its authors, its readers, and itself

There’s a growing awareness that using computer algorithms to handle decisions — from debt assessment or even being fired to self-driving cars — can have unexpected and even disastrous effects. Around October 2017, Amazon joined those unfortunate ranks.
Sounds scary? It is, when you dig below the surface a little.
Over a million new books are now published each year. Over half of those are in English.
I started publishing in December 2015, using Amazon exclusively for my eBook editions. Sales and reviews were low but showed a steady growth. My approach was to concentrate on writing each new book and do the bare minimum on the marketing side.
For almost two years that seemed to be working — until October 2017.
At that point, sales plummeted. Had I done something, or said something to upset people? Highly unlikely, since I’d been silently working on the current book. I assumed my silence was the cause. But after discussing sales with a few other authors who used Amazon ads and active marketing, I changed my mind. They told me that from around October 2017, they found if they pay $X to Amazon for ads, they came out ahead by about 10%.
What happened? What follows is my best guess. There are three parts to it: book discovery, Kindle Unlimited, and unscrupulous people.

Book Discovery

Readers can discover books on Amazon through various searches, but in addition there are three important ways Amazon promotes books:
Recommendations

1. Emails from Amazon to you suggesting books you might like.
   
2. Suggestions for other books based on your searches, on Amazon web pages.
   

Note: included in these recommendations are books that have paid to be promoted.
Best-seller lists. In each category, a list of the books that have earned the most revenue in the last 48 hours.
Popularity lists. Like the best-sellers, but across the last 30 days.

Kindle Unlimited

Readers who pay the monthly subscription fee can borrow books from Kindle Unlimited (KU) at no extra cost. Some of the subscription fees go into a pot of money paid to the authors.
This ‘Cockygate’ article by the Verge notes: “There are over 5 million books available via Kindle, with over a million books available on the Kindle Unlimited system. Amazon is currently paying out approximately 20 million dollars every month to authors on Kindle Unlimited.”
U$20M lying on the table: does that sound like a tempting target for unscrupulous people?
At first, Amazon divided up the pot based on number of books sold. Absurdly short ‘books’ started appearing. Amazon changed the algorithm to count ‘pages read’ instead of ‘books read’. Scammers started stuffing books full of ‘bonus material’ to dramatically boost their number of pages.
It’s been an arms race between scammers trying to grab money unfairly from the pot, and Amazon trying to counter them. This article by Derek Haines argues Amazon’s algorithms can’t count pages or words, but he goes on to add:
“KU scammers and ebook stuffers are not the real issues here.
[…] Until Amazon can find a better system to track and record KU ebook reading correctly, the scammers will keep finding ways to make money, and honest authors will lose out.
Or be terminated.”
But wait, there’s more…!

How recommendations used to work

Before Oct 2017, Amazon’s algorithms recommended books based on some score related to average review rating, the sales rate, and purchase by other people of similar books. (“People who bought TitleX — as you did — also often bought TitleY.”)
I think this system is still used for the 30 days after a book is released, perhaps extended if the book hits a best-seller list or while it is earning high revenue.

How they work, post Oct 2017

The only books being promoted were those from paid ads. At first, I guessed that Amazon had changed its algorithms to only promote advertised books. There are so many authors publishing, Amazon can profit from authors buying ads.
Amazon don’t disclose how their algorithms work, and rightly so. They’re in a war with those who try to game the system to make money. I have a lot of sympathy with Amazon and what they face.
But since October 2017, the KU pool of money has been paying large sums of money (perhaps most of it) to some unscrupulous ‘authors’. I say ‘authors’ because some are not authors: they are business people who pay ghost writers for content.
The result is, only books with paid ads are being promoted.

Unscrupulous People

David Gaughran’s Kindle Unlimited -- A Cheater Magnet and Kayleigh Donaldson’s Book Stuffing, Bribery and Bullying articles were eye-openers for me — I recommend them. The problem is that payments to unscrupulous people has a terrible effect on honest authors, harms readers, and has started damaging Amazon’s brand (and, I assume, the revenue from self-published books).
The effect on the readers is the smallest: the unscrupulous people can’t afford to annoy them too much, since readers determine the KU payouts. But even for readers, it’s making it harder to find and read books they’ll enjoy.
The unscrupulous people spend big, buying Amazon ads to turn a good profit, paying for those ads from the profits. If collectively they spent 18 million dollars for a 19 million dollar payout from KU, they’d be happy — and hey, that even leaves a million dollars to be divided amongst the remaining 500,000 authors! <sarcasm off/>
The greatest harm is being done to the honest self-published and indie authors. Like most skills, writing follows a Bell curve. A minority are really good; a minority are really bad; and there’s a big spread in the middle between those extremes. (The true picture is more complex, because everyone’s taste differs. But that’s not important for this argument.)
But most good and great writers have no interest or skills in marketing, nor in spending time and effort on it. So most authors can’t compete when it comes to marketing.
If you’re an author selling via Amazon and your readers have declined and your income has plummeted, take heart at least in knowing it’s likely not any indicator of how good your book is. Your small voice is being drowned out by the people gaming the Amazon system. The unscrupulous are producing ten or fifty books to your one, and paying for vast numbers of Amazon recommendations of their books, rather than a meagre few algorithmically-generated recommendations that yours are getting.
Writing a book is hard work. A lot of writers manage one. Producing a book proves to an author they can do it. But without a lot of luck, or a big and skilful marketing effort, few readers will discover the book, so the return is low in terms of money and in appreciation. And with Amazon’s ineffectual response to the cheaters, a lot of new authors will try, but won’t find out if they are great or even good. Unless new authors stumble over this article or one like it, they won’t know the failure may well have been Amazon’s, not theirs.
On top of that, the Amazon algorithms only seem to promote books for the first 30 days. This of course encourages writers to produce shorter books and pressures them to cut corners. For the unscrupulous, it actually plays into their hands since producing content quickly is something they can pay or cheat to produce — they just need to keep one step ahead of Amazon’s lacklustre attempts to rein them in.
By inducing many of the good and some of the best new authors to give up, while still ‘feeding the scammers’, over time the overall quality of Amazon’s book marketplace will decline. At first, not massively. Just a little. But the trend will continue, and in the long term will make Amazon the place for second-rate books which are clearly lower quality than those from other publishers.
The Amazon book marketplace will swell with the people who pay for the ads, and the people with good marketing skills. There’ll be some very good books in there, but on average, the rewards will go to the best marketers and to the scammers, not to the best writers.

How Amazon Could Fix This

It’s in Amazon’s interest to fix this, despite what some cynical people argue. Like other observers, I believe the problem lies as much with Amazon as with the scammers. Amazon seem determined to address the problems with technology (algorithms), not people, and I can see the sense of that: the solution must scale to handle millions of books, hundreds of thousands of authors, as well as defend against sets of active and inventive scammers trying to snatch an unfair share of the money available.
Taken together, this also means that until Amazon develop a solution to the problem, the company appears disinterested and unsupportive, damaging their brand even as the quality of their content declines.
So here’s my set of suggestions for Amazon (the first two of these I owe to a friend, Geoff Dash).
Follow the money. The problem of the scammers is proportional to the amount of money they’re taking. So audit the highest-earning authors in order of the revenue they earn. Examine their books to ensure they’re meeting Amazon’s Terms of Service rather than gaming the algorithms. Add legitimate authors to a register of Verified authors (and even mark their books as such). Add scammers to a list of banned authors and remove them from Amazon.
Suspect the absurdly productive. Amazon could reduce the number of authors they need to Verify by examining them in order of productivity: the authors producing the most books per year (or, per month).
How to Verify. It doesn’t have to be done manually, although the above two principles make that feasible. Instead, deep learning algorithms could be trained with examples of scam content and genuine content. Amazon could use the Mechanical Turk to help examine the books selected for audit. It could build a set of trusted Verifiers. It could use some of its own people to examine the books and explain to the algorithm developers what patterns and qualities to look for.
Only Count Content Once. Use its internal plagiarism detecting algorithms to identify content so that when a reader ‘reads’ a book, that content is only counted once. I imagine something like MD5 markers for a book, stored in a reader’s account. That information could even improve the user experience, by offering the reader a chance to skip the previously-read content.
Be more Transparent. It’s fine to have algorithms making decisions for you, and you need to keep their workings secret to make things harder for the scammers. But have a good and fair system for resolving disputes. Amazon needs to do this because its record of hurting good guys with bad algorithms has damaged its reputation in this respect.
Algorithmically select for quality. By this I mean, engineer algorithms that over time will encourage high quality work by offering reasonable revenue to the author. At the least, stop using algorithms that encourage low quality. The current system that only seems to recommend books published in the last 30 days encourages ‘publish fast’ over ‘publish high quality’. Blocking reviews from professional bloggers and reviewers who are gifted with so many books they don’t spend over $50, is another algorithm that does not select for quality.

What Will I Do?

At present, I see no value in being in Kindle Unlimited, and little value in selling your books through Amazon at all. Being exclusive with Amazon is worthless after the first three months. I recommend anyone who has currently chosen exclusivity with Amazon, to terminate that exclusivity. You may need to wait three months for your current exclusivity period to end, but when your sales are zero there’s no difference in revenue between 35% or 70%. Start publishing your books elsewhere, where you won’t be drowned out by ads and scammers.
If enough authors take this step, it will also send a signal to Amazon. Hopefully, such a market-savvy company will recognise and react to that. Certainly, from reports I’ve read, Amazon doesn’t appear to react to any other signals.
I'm quoting from the Verge’s ‘Cockygate’ article, in which Zoe York (romance author) notes:
“All of these people, without exception, all of them, publish exclusively on Amazon,” she said of the book-stuffers. “This is not a problem on iBooks, on Kobo, on Barnes and Noble, or on Google Play, which are the other major ebook retailers.”


Amazon did a wonderful thing in basically creating and nurturing the ebook marketplace, and through that, stimulating the self-published marketplace. It would be sad if it misstepped now and its know-how and support became irrelevant.
Amazon now rewards good marketers (honest ones, but also the scammers), not good authors. That needs to be fixed, or it will likely lead to the collapse of Amazon as a reputable self-publisher.