Literary genres weren’t invented by writers. They were invented by publishers. Writing is notoriously hard to categorize. Each book is unique, even books written by the same author. Publishers’ marketing departments hate this. Promote each book as a unique entity? That’s no way to do business.
So publishers identify niches, big groups of potential readers with similar interests, and market to them. Much easier, much more efficient, and (from the marketers’ perspective) much more effective.
At first the boundaries between niches are flexible, barely perceptible even. But with time they harden into walls. The niches become genres, and the genres subdivide into subgenres. The walls get higher, so that readers can barely see over them.
Reading Ray and Lorna Coppinger’s book Dogs: A Startling New Understanding of Canine Origin, Behavior, and Evolution, I got the idea that the evolution of literature has a lot in common with the evolution of dogs.
The Coppingers believe that canids domesticated themselves (and evolved into dogs) after humans started settling into stationary villages, which featured on their outskirts stationary dumps — good foraging for those canids who didn’t mind being that close to people. These “village dogs,” both in history and in the present, developed certain common characteristics, e.g., a medium size that was big enough to defend itself but small enough to thrive on the food available. Physical differentiation started to happen when some humans figured out that dogs could be useful at certain tasks, such as herding or guarding livestock and pulling sleds.
“Breeds” as we know them, however, are relatively recent developments. Until a couple of centuries ago, a retriever was a dog that retrieved well — not necessarily a dog with retrievers on both sides of its pedigree. Whether a dog could retrieve well depended partly on its parents (did it have the right physical characteristics for the job?) and partly on how it was raised and trained. A Labrador retriever pedigree alone does not a good retriever make. The modern emphasis on “pure” breeds (meaning the stud books are closed, meaning genetic diversity takes a wallop), especially in the show ring, tends to divorce function from appearance and to focus heavily on the latter.
Literary genres are like breeds — of relatively recent development, especially the notion that there are clear lines between them and everything has to fit into one category. “Literature” is more like those village dogs of indeterminate breed: it adapts to the climate and food sources available, and maybe it looks a little like this, a little like that, but you can’t say for sure that it’s a beagle or a foxhound (or a mystery or a romance). When you’re trying to tell a story, you scavenge and steal from whatever’s in the vicinity and if it works you keep it.
All of which is not to say that genres aren’t useful to writers. Faced with a whole boundless ocean of possibilities, it’s easy to choke. Why not focus on the weather and currents, the flora and fauna, of a particular inlet or harbor or archipelago? By all means go ahead. Each genre has its tropes and conventions. Because readers are familiar with them, you don’t have to justify, say, the dead body that turns up in chapter 1 or the FTL (faster-than-light) starship that enables your characters to get from one planet to another. You can devote your writerly attention to other things.
Just keep in mind that many stories worth telling don’t fit neatly, or even messily, into one genre or another. In the attempt to squeeze them in, sacrifices have to be made. Genetic diversity may be lost. The end result might be a dog that looks sharp in the show ring but can’t do the job its ancestors did.