There’s a growing awareness that
using computer algorithms to handle decisions — from
debt
assessment or even being fired to self-driving
cars — can have unexpected and even disastrous effects. Around
October 2017, Amazon joined those unfortunate ranks.
Sounds scary? It is, when you dig
below the surface a little.
Over a million new books are now
published each year. Over half of those are in English.
I started publishing in December
2015, using Amazon exclusively for my eBook editions. Sales and reviews
were low but showed a steady growth. My approach was to concentrate
on writing each new book and do the bare minimum on the marketing
side.
For almost two years that seemed
to be working — until October 2017.
At that point, sales plummeted.
Had I done something, or said something to upset people? Highly
unlikely, since I’d been silently working on the current book. I
assumed my silence was the cause. But after discussing sales with a few
other authors who used Amazon ads and active marketing, I changed my
mind. They told me that from around October 2017, they found if they
pay $X to Amazon for ads, they came out ahead by about 10%.
What happened?
What follows is
my best guess. There are
three parts to it:
book
discovery,
Kindle
Unlimited, and
unscrupulous people.
Book Discovery
Readers can discover books on Amazon through
various searches, but in addition there are three important ways
Amazon promotes books:
Recommendations
- Emails from Amazon to you suggesting books you
might like.
- Suggestions for other books based on your
searches, on Amazon web pages.
Note: included in these recommendations are books
that have paid to be promoted.
Best-seller lists. In each category, a
list of the books that have earned the most revenue in the last 48
hours.
Popularity lists. Like the best-sellers,
but across the last 30 days.
Kindle Unlimited
Readers who pay the monthly subscription fee can
borrow books from Kindle Unlimited
(KU) at no extra cost. Some of
the subscription fees go into a pot of money paid to the authors.
This
‘Cockygate’
article by the Verge
notes: “There are over 5 million books available via Kindle, with
over a million books available on the Kindle Unlimited system. Amazon
is currently paying out approximately
20
million dollars every month
to authors on Kindle Unlimited.”
U$20M
lying on the table: does that sound like a tempting target for
unscrupulous people?
At
first, Amazon divided up the pot based on number of books sold.
Absurdly short ‘books’ started appearing. Amazon changed the
algorithm to count ‘pages read’ instead of ‘books read’.
Scammers started stuffing books full of ‘bonus material’ to
dramatically boost their number of pages.
It’s
been an arms race between scammers trying to grab money unfairly from
the pot, and Amazon trying to counter them. This article by Derek
Haines argues Amazon’s
algorithms can’t count pages or words,
but he goes on to add:
“
KU
scammers and ebook stuffers are not the real issues here.
[…] Until Amazon
can find a better system to track and record KU ebook reading
correctly, the scammers will keep finding ways to make money, and
honest authors will lose out.
Or
be terminated.”
But wait, there’s more…!
How recommendations used to work
Before Oct 2017, Amazon’s
algorithms recommended books based on some score related to average
review rating, the sales
rate, and purchase by other people of
similar books. (“People who bought
TitleX — as you did —
also often bought
TitleY.”)
I think this system is still used
for the 30 days after a book is released, perhaps extended if the
book hits a best-seller list or while it is earning high revenue.
How they work, post Oct 2017
The only books being promoted were those from paid ads. At first, I guessed that Amazon
had changed its algorithms to only promote advertised books. There are so many authors publishing, Amazon can profit from authors buying ads.
Amazon don’t disclose how their
algorithms work, and rightly so. They’re in a war with those who
try to game the system to make money. I have a lot of sympathy with
Amazon and what they face.
But since October 2017, the KU
pool of money has been paying large sums of money (perhaps most of
it) to some unscrupulous ‘authors’. I say ‘authors’ because
some are not authors: they are business people who pay ghost writers for content.
The result is, only books with paid ads are being promoted.
Unscrupulous People
David Gaughran’s
Kindle
Unlimited -- A Cheater Magnet and Kayleigh Donaldson’s
Book
Stuffing, Bribery and Bullying articles were eye-openers for me — I
recommend them. The problem is that payments to unscrupulous
people has a terrible effect on honest authors, harms readers, and
has started damaging Amazon’s brand (and, I assume, the revenue
from self-published books).
The effect on the readers is the
smallest: the unscrupulous people can’t afford to annoy them too much,
since readers determine the KU payouts. But even for readers, it’s
making it harder to find and read books they’ll enjoy.
The unscrupulous people spend big,
buying Amazon ads to turn a good profit, paying for those ads from
the profits. If collectively they spent 18 million dollars for a 19
million dollar payout from KU, they’d be happy — and hey, that even leaves a
million dollars to be divided amongst the remaining 500,000 authors! <sarcasm off/>
The greatest harm is being done to
the honest self-published and indie authors. Like most skills,
writing follows a Bell curve. A minority are really good; a minority
are really bad; and there’s a big spread in the middle between
those extremes. (The true picture is more complex, because
everyone’s taste differs. But that’s not important for this
argument.)
But most good and great writers
have no interest or skills in marketing, nor in spending time and
effort on it. So most authors can’t compete when it comes to
marketing.
If you’re an author selling via
Amazon and your readers have declined and your income has plummeted,
take heart at least in knowing it’s likely not any indicator of how
good your book is. Your small voice is being drowned out by the
people gaming the Amazon system. The unscrupulous are producing ten
or fifty books to your one, and paying for vast numbers of Amazon
recommendations of their books, rather than a meagre few
algorithmically-generated recommendations that yours are getting.
Writing a book is hard work. A
lot of writers manage
one. Producing a book proves to an
author they can do it. But without a lot of luck, or a big and
skilful marketing effort, few readers will discover the book, so the
return is low in terms of money and in appreciation. And with
Amazon’s ineffectual response to the cheaters, a lot of new authors
will try, but won’t find out if they are great or even good.
Unless new authors stumble over this article or one like it, they
won’t know the failure may well have been Amazon’s, not theirs.
On top of that, the Amazon
algorithms only seem to promote books for the first 30 days. This of
course encourages writers to produce shorter books and pressures them
to cut corners. For the unscrupulous, it actually plays into their
hands since producing content quickly is something they can pay or
cheat to produce — they just need to keep one step ahead of
Amazon’s lacklustre attempts to rein them in.
By
inducing
many
of the good and some of the best new authors to
give up, while still ‘feeding
the scammers’, over time
the overall quality of Amazon’s book marketplace will decline. At
first, not massively. Just a
little. But the trend
will continue, and
in the long term will make
Amazon the place for second-rate
books which are
clearly lower quality than those
from other publishers.
The
Amazon book marketplace will swell with the people who pay for the
ads, and the people with good marketing skills. There’ll be some
very good books in there, but on average, the rewards will go to the
best marketers and to the
scammers, not to the best
writers.
How Amazon Could Fix This
It’s in Amazon’s
interest to fix this,
despite what some cynical
people argue. Like
other observers, I believe the problem lies as much with Amazon as
with the scammers. Amazon seem determined to address the problems
with technology (algorithms), not people, and I can see the sense of
that: the solution
must
scale to handle millions of books, hundreds of thousands of authors,
as well as defend against sets of
active and inventive scammers trying to snatch
an unfair share of the money available.
Taken
together, this also means that until Amazon develop a solution to the
problem, the company
appears disinterested and unsupportive, damaging
their brand even as
the quality of their content declines.
So
here’s my set of suggestions for Amazon (the first two of these I
owe to a friend, Geoff Dash).
Follow the money. The problem
of the scammers is proportional to the amount of money they’re
taking. So audit the highest-earning authors in order of the revenue they earn.
Examine their books to ensure they’re meeting
Amazon’s Terms of Service rather
than gaming the algorithms.
Add legitimate authors to a register of Verified authors (and even
mark their books as such). Add
scammers to a list of banned authors and remove them from Amazon.
Suspect the absurdly productive.
Amazon could reduce the number of authors they need to Verify by examining them
in order of productivity: the
authors producing the
most books per year
(or, per month).
How to Verify. It
doesn’t
have to be
done manually, although the above two principles make that feasible.
Instead, deep learning
algorithms could be trained with examples of scam content and genuine
content. Amazon could use the Mechanical Turk to help examine the
books selected for audit. It could build a set of trusted Verifiers.
It could use some of its own people to examine the books and explain
to the algorithm developers what patterns and qualities to look for.
Only Count Content Once. Use
its internal plagiarism detecting algorithms to identify content so
that when a reader ‘reads’ a book, that content is only counted once. I
imagine something like MD5 markers for a book, stored in a reader’s
account. That information could even improve the user experience, by offering the reader a chance to skip the previously-read content.
Be more Transparent. It’s
fine to have algorithms making decisions for you, and you need to
keep their workings secret to make things harder for the scammers.
But have a good and fair system for resolving disputes. Amazon needs
to do this because its record of hurting good guys with bad
algorithms has damaged its reputation in this respect.
Algorithmically select for quality. By
this I mean, engineer algorithms that over time will encourage
high quality work by offering reasonable revenue to the author. At the least, stop using
algorithms that encourage low quality. The current system that only
seems to recommend books published in the last 30 days encourages
‘publish fast’ over ‘publish high quality’. Blocking
reviews from professional bloggers and reviewers who are gifted with
so many books they don’t spend over $50, is another algorithm that does not select for
quality.
What Will I Do?
At present, I see no value in being in Kindle Unlimited, and little
value in selling your books through Amazon at all. Being
exclusive with Amazon is worthless after
the first three months. I
recommend anyone who has currently chosen exclusivity with Amazon, to
terminate that exclusivity. You may need to wait three months for
your current exclusivity period to end, but when your sales are zero
there’s no difference in revenue between 35% or 70%. Start
publishing your books elsewhere, where you won’t be drowned out by
ads and scammers.
If
enough authors take this step, it will also send a signal to Amazon.
Hopefully, such a market-savvy company will recognise and react to
that. Certainly, from
reports I’ve read, Amazon doesn’t appear to react to any other
signals.
I'm quoting from
the
Verge’s ‘Cockygate’ article,
in which
Zoe
York (romance author) notes:
“All
of these people, without exception, all of them, publish exclusively
on Amazon,” she said of the book-stuffers. “This is not a problem
on iBooks, on Kobo, on Barnes and Noble, or on Google Play, which are
the other major ebook retailers.”
Amazon
did a wonderful thing in basically creating and nurturing the ebook
marketplace, and through that, stimulating the self-published
marketplace. It would be sad if it
misstepped now and its
know-how and support became irrelevant.
Amazon
now rewards good marketers (honest
ones, but also the scammers),
not good authors. That needs to be fixed, or it will likely lead to the collapse of
Amazon as a reputable self-publisher.
