The digital revolution is marvellous but it is expensive; publishing needs a visionary for the digital age, says Gordon Johnson

Jeremy Lewis's sparkling life of Allen Lane is published as part of the 70th-anniversary celebrations for Penguin books. It is immensely readable and full of good anecdotes. Lane was an inspired publisher. He may not have bothered much about reading books, but his instinct told him that the 1930s offered a whole new opportunity to sell them. The postwar rise of literacy in Britain had created a demand waiting to be gratified.

Lewis argues that Lane's genius lay in seeing how to offer books of quality at a low price. Penguins were carefully chosen, imaginatively packaged and sold for less than the cost of a packet of cigarettes. Part of Lane's strategy was to foster an addiction to books - his books. Readers became hooked and looked out each month for the next batch of Penguins, which (along with Pelicans, Puffins, Peregrines and so on, all part of the same brand) came to embrace reprinted fiction, new writing, classics, crime, non-fiction, current affairs, books that taught you things, poetry, music and art. It was stylish and fashionable to be a Penguin reader. More seriously, the cultural habits of a lifetime and the intellectual outlook of a generation were formed by the Penguin publishing programme. The Penguin phenomenon demonstrates to publishers that knowledge in itself is useless unless disseminated, and that how to do that is a special art. Lewis is a sympathetic biographer, but his book also gives an excellent account of what publishing as a whole was like in the 1930s and of how it changed in the following three decades. Lane was nothing if not a mercurial character; publishing is nothing if not an inconstant business.

John Thompson, a professor of sociology at Cambridge University, is also a publisher, being one of the brains behind Polity, a small publishing house that has carved out an influential position in the harder social sciences.Thompson's direct, practical knowledge, coupled with support from the Economic and Social Research Council to undertake a three-year research project on academic publishing, has produced a fascinating, if rather long and inconclusive, study of the contemporary academic publishing world in the UKand the US. Furthermore, it draws on the experience of a fair cross-section of top publishers. While it is fun (and in some cases possible) to guess who said what in the many anonymous quotations taken from Thompson's survey material, these give such extra interest that the book's value would have been further enhanced by revealing the identity of his informants.

ÁñÁ«ÊÓƵ

The rise of literacy, which made Lane and Penguin wealthy, led after the Second World War to a golden age for commercial publishers and for university presses. University teachers, who previously had felt no particular urge to write books or articles, were first attracted, then induced and finally required to publish research. Others found that there were lucrative opportunities to write textbooks even if, or perhaps particularly if, this required talents that lay more in the simplification and exposition of leading research than the creation of it. Publishers fell over themselves to sign them up. The good times lasted, Thompson suggests, almost to the end of the 20th century (I would say that chilly breezes, later to reach gale force, had started to blow across the landscape by the mid-1970s). The demand for monographs collapsed, and scholarly books became too expensive for individuals. Even university libraries, pressed to buy multiple copies of essential textbooks and pay outrageous subscriptions for journals and electronic media, were buying fewer new books and hardly any old ones.

The knock-on effects have been considerable: scores of independent publishers are now subsidiaries of three or four great conglomerates, with a serious loss of diversity in publishers' lists. Academic books, with their low sales and small profit margins, have been squeezed out. Journal publishing has fallen into the hands of two or three giants in the field, which, driven by escalating costs and the need for profit, have held universities to ransom with annual increases in subscription costs at double or triple the level of retail price inflation. And whereas once upon a time a good textbook in the sciences, law or engineering might hold the field for several years - even decades - and thus produce a fair return on the original (heavy) investment, to hold market share now demands constant and costly revision.

ÁñÁ«ÊÓƵ

The university presses provide some relief, for they still publish academic work of high quality. But they have not been immune to the general changes. Even the largest and most prestigious university presses were always small players in the industry at large, and university presses have in general been forced to depend more on subsidy (either in cash from their parent university - the position of most American presses), or to refocus severely the nature and scale of their publishing activity. To sustain monograph publishing and to maintain their journals, even the university presses of Oxford and Cambridge have become more selective in what they agree to publish. They have diversified their lists, published more textbooks and put effort into searching for that elusive scholarly book that might be read outside academe.

There is, therefore, a crisis of sorts in academic publishing, felt rather than properly understood. It has long been a common belief that this crisis would be overcome by the application of digital technology to the problems of publishing: the contemporary equivalent of Lane's spotting that a cheap, stylish book would sell by the thousands. This time round, almost everyone said that the book and the printed journal had had their day; material would be published electronically and made readily available to all via laptop computers and mobile phones. Knowledge would become "democratised"; freely accessible at virtually no expense to the consumer.

Thompson's research begins to show why this has not happened and why the crisis shows no sign of abating. To begin with, the fact that something is technologically possible (or will become so without much more effort) does not mean that it meets a social need. In an analysis that is as compelling as it is thorough, and exploiting the tools of the sociological trade in ways that are accessible to the lay reader, Thompson shows us just how complex publishing is for the academy. It is not enough to write a good book or journal article - you have to get it to the reader. How this is done demands an understanding of what readers want and what they will tolerate.

The market for academic books is exceptionally divided in this regard, falling into many different segments. A failure to grasp this means that technological advances may not have the expected impact. Publishers must understand the sociological and political context of how knowledge is received and used. What Thompson concludes is that the market, that is the readers, will take advantage of new ways of receiving knowledge but without necessarily forsaking the old. A sustained piece of writing will be read in hard copy and not on screen. Yes, it is convenient to find an article via the laptop at the laboratory bench and perhaps even play with the equations online, but one may also download it for considered study and one will expect the library to be able to produce it in future in print, should its electronic form become inaccessible. Furthermore, technological change may permit the subversion of carefully balanced publishing structures: students will not necessarily feel obliged to buy textbooks if they can photocopy the textbook for a small cash outlay or download from the web, in effect free of charge, the pages they need.

ÁñÁ«ÊÓƵ

The digital revolution has certainly transformed some processes of publication: printing with metal on paper has gone for ever, and the whole complex of talents and skills that went into the printed page are disappearing. Some costs of publication have been dramatically cut over the past 20 years: the expense of producing a page of a science journal is down from more than ?20 with the old technologies to ?2 or ?3 with the new. Similarly, the "printing" cost per page of a conventional book is now a very small proportion of the cost of putting the book into a shop.

Of course, some of these savings in costs have simply been redistributed - to authors, for example, who must now be their own typesetters, designers, copy editors and proofreaders - with a definite loss of quality in consequence. But what is remarkable is that there has been such resistance to relying on electronic publication only. Publishers may be doing less "printing", but others are doing it for them. Just as it has been calculated that the "paperless" office consumes 15 per cent more paper than the traditional office, so the sale of paper to individuals through supermarkets for printing at home has more than compensated for the drop in the sale of paper to the printing industry.

Moreover, the digital revolution is both marvellous and expensive. It is amazing that diagrams in a scientific paper can now be manipulated by readers, and that textbooks come with electronic materials that keep it constantly up to date. Having experienced such benefits, the academic world wants them as standard. But it does not have the resources to pay for them. Everyone is becoming accustomed to "free" access to knowledge, and there is no real understanding of the costs involved in its creation and management. Some of the biggest publishing conglomerates have withdrawn, at least for the moment, from full-scale electronic publications, in some cases having spent tens of millions with no prospect of a viable return. The issues of how to assure the quality of information, how to retain ownership of the intellectual property rights and how to conserve the information beyond a year or so after its generation are legion, and they have barely begun to be tackled.

So it is no wonder that the digital revolution has neither killed the book nor saved the monograph. Even in the digital age, the publishing industry will continue to need its eccentric Allen Lanes who will sense, rather than know, the way things are and are going and who will bring imagination and creativity to the publishing business.

ÁñÁ«ÊÓƵ

Gordon Johnson is president of Wolfson College, Cambridge, and chairman of Cambridge University Press.