In the past, the Royal Academy has often viewed membership of its friends as not much more than a season ticket to exhibitions: not in any way a loyalty card or a badge of philanthropic association.

But nowadays we want friends to come to the Royal Academy even when there isn’t an exhibition – to do their Christmas shopping, have lunch, meet, enjoy a small, therapeutic break from the hurly-burly of a day out in the “Great Wen”, perhaps try one of the cocktails in the bar, and, in the summer, sit out in the small garden, which lies between us and Albany, and which has been sensationally well designed by Tom Stuart-Smith.

There is a deeper level of ideas and aspirations involved. We live in an increasingly competitive marketplace, where arts institutions in London are competing for the love and loyalty of their visitors. Of course, much of this loyalty is inspired by the art which is on display. But it is not just a matter of the art. Every institution has its own character and inspires a distinctive set of subliminal associations.

As an art institution, the Royal Academy needs to pay close attention to all aspects of our public programme, to the quality and character of our exhibitions, and how they interpret their subject.

But I also believe that we need to be attentive to what people eat, what they sit on, what they want to buy. Loyalty to an institution is inspired not just by its public programme, but by its psychological appeal, whether it feels comfortable or democratic or intellectually challenging, or a combination of all three.

As a result of this thinking about who we are and what we want the Royal Academy to be, I have been preoccupied over the last six-and-a-bit years, since being appointed as secretary and chief executive, in the development of the so-called Keeper’s House – a project which has been long in the gestation, problematic in some aspects of its implementation, but which has succeeded beyond our wildest expectations in creating high-quality facilities for academicians to meet clients, and our friends, members and patrons to recuperate after seeing an exhibition.

It’s a weird mixture of domesticity and internationalism, ad hoc-ery and modernism

Sandy Wilson, the architect of the British Library (so unfairly reviled for the project) spent the last six years of his life devising a master plan for the development of the Royal Academy’s large site, which now stretches more or less all the way from Piccadilly in the south, northwards through the old 17th-century mansion of Burlington House, to the old University of London headquarters in Burlington Gardens. The total area is said to be as large as the footprint of the British Museum. Wilson proposed developing the old Keeper’s House, which lies to the right of the main façade of Burlington House.

It was originally designed by Sydney Smirke, son of the architect of the British Museum and responsible for its round reading room, as the domestic residence of the Keeper of the Royal Academy, who was required not only to supervise the teaching of the students in the Royal Academy Schools, but also to lock up at night, feed the cat and be the only resident on site. The Keeper had a small drawing room on the ground floor, a studio with north-facing light on the top floor and a private staircase for models to enter without being seen from the garden.

Wilson oversaw plans which would make the Keeper’s House into private, club-like facilities for the Friends of the Royal Academy in order to secure their loyalty, give them somewhere to sit and have a cup of tea, and, at the same time, provide better quality space and facilities for the Royal Academicians, who have not always been well looked after by the institution, which was founded, and still exists, to represent their interests.

Sadly, Sandy Wilson died not long before I started as secretary and chief executive. His role in the detailed design of the Keeper’s House was inherited by the firm M.J. Long, established by his widow, and their principal assistant, Rolfe Kentish. Long and Kentish have been responsible over the years for a great deal of relatively small-scale, thoughtful studios for artists, including the studio complex in Hammersmith occupied by Sir Peter Blake and Paul Huxley and, more recently, the renovation of the studios on Porthmeor Beach in St Ives, once occupied by the likes of Patrick Heron.

They have also worked on a number of restaurant projects, including the original Ivy and, more recently, the renovation of Quo Vadis. Much of the quality of the final result is owing to their care and attention (as well as that of Julian Harrap as our conservation architect) to the detail of the project, including retaining the original, rough-textured timber boarding of the so-called Architecture Room, which was added to the main set of galleries by Norman Shaw in the early-1880s, inserting a glass lift into the space between the Keeper’s House and the Schools, putting a round port hole into the north wall of the Architecture Room, and retaining a sense of an archaeological dig in the lavatories in the basement.

At a relatively late stage of the project, it was decided that the interior design of some of the key rooms should be done by Sir David Chipperfield, who has inherited the mantle of Sandy Wilson as master planner for the Royal Academy as a whole. Although he was a specialist in many small-scale projects in the 1980s – I still treasure a copy of the 1980s booklet produced by 9H Gallery on the work of Chipperfield, Rick Mather, Eric Parry and Stanton Williams, which includes the shop design Chipperfield did for Issey Miyake in Sloane Street in 1983 – he has since been much better known for large-scale, museum projects, including, most obviously, the Neues Museum in Berlin and, in England, Turner Contemporary in Margate and Hepworth in Wakefield.

I like to think that he took the project as a challenge to return to his roots as someone with a deep sensibility for interior spaces – lighting, colour, textures and detailing – not the normal armoury of a modernist architect.

What Chipperfield has added to the project is more than a dash of cosmopolitan chic. The basement dining room, converted out of some subterranean offices, was always going to be a difficult space, with no access to natural daylight. He has made it into two discreet rooms, each rather intimate in feel, lined with green billiard baize, with classic Goteborg chairs, designed by Gunnar Asplund, hung with small, antique casts given to the Royal Academy’s collection by Sir Thomas Lawrence, and lit by small sphere lights with gilt reflectors, specially designed by Chipperfield in collaboration with Mario Nanni and manufactured by Viabizzuno.

On the other side of the basement corridor is a low-ceilinged bar, bright red, described by Apollo magazine’s Muse Room as “like being in a bento box”, which is possibly apt since Chipperfield spent the late-1980s in Tokyo. On the ground floor, the original Friends’ Room has had a makeover, keeping the fine proportions of the original room and with a floor of Danish oak boards, a space in which to enjoy a cappuccino. And beyond, overlooking the little garden, sandwiched between the Royal Academy and Albany, is a small sitting room, with red chairs supplied by Azucena, a small Milanese design firm, which look as if they have been made for dwarves, but are surprisingly comfortable.

There are two other key elements to making the experience feel domestic and not too institutional. The first is the way the pictures are hung, with photographs of many of the current Academicians hung up the main staircase, and large paintings by Christopher Le Brun, Ian McKeever, Anthony Green and Humphrey Ocean in the Academicians’ Room upstairs.

The second is the quality of the food. This has been overseen by Oliver Peyton, who has brought in his best and most creative chef, Ivan Simeoli, who was previously in charge of the kitchens at the Wallace Collection, to create menus of inventive nouvelle cuisine, not too heavy and made from the freshest ingredients, including fresh partridge and venison, wild Cornish sea bass, and damsons picked in Epping forest. It’s a feast of the senses.

I do not feel that the Keeper’s House was methodically and meticulously planned to be exactly what it has turned out to be. But, in practice, I like to think that it encapsulates the odd cacophony of the Royal Academy’s core values – its weird mixture of domesticity and internationalism, ad hoc-ery and modernism, anti-metropolitan friends and Tracey Emin, the old red timber boards bequeathed by Norman Shaw combined with a brand new bar.

Where else would you expect to find Australian palm trees right in the heart of old London? Or be able to look out from a balcony on to a red Michael Craig-Martin fork planted in ferns?