A month ago I was sitting with my beloved on the subway, and at the end of our row of seats there sat an elderly gentleman, with a paunch, and a beard as long as Father Christmas's, but stained with nicotine, like dirty snow. He was wearing maybe twenty different layers of clothing, and balanced before him, on his walker, was a walking-stick and many, many plastic bags of great antiquity. No-one else in the carriage was rude enough to stare (Americans are a very polite people, and as a rule, very respectful of each other's space and nuttiness), but as I watched, the carriage was filled with the sound of a strident ring-tone (Lady Gaga, I believe) and from his pocket, Daddy Christmas pulled an iPhone 5. A loud, one-sided conversation followed. "No. Jeez, we just left Brooklyn. I'll be an hour. Yeah an hour. We just left Brooklyn. Jeez, yeah, an hour. You get the car to meet me. Yeah, an hour." Phone back into pocket. Two minutes later my beloved, who is the most wonderfully courteous American, but who humours my crude Old World nosiness, whispered "Look what he's got in his hand now." Hoping very sincerely indeed that I would not regret this, I turned my head. Father Christmas was now holding in one hand, and counting through, with great concentration, a wodge of dollars maybe 3 inches thick.

The point of all this is that the greatest spectacle, the greatest display, the greatest work of art in all New York is New York itself, and it never stops performing. If you find yourself there, don't forget it. It's a city you should walk with your eyes peeled. If you don't, you're missing half the show.

But if you want to be able to feast your eyes legitimately, as it were, New York is equally the place. The museums here are among the finest in the world, indeed in the Metropolitan Museum of Art they have what is probably the finest of them all. If you did a mash-up of the top ten museums in Europe – the Louvre, the British Museum, the Prado, the National Gallery, the V&A, Topkapi, maybe, plus the Egyptian Museum in Cairo, you might have one half the number of masterpieces on display here. Some of the objects in the Met's collection will simply leave you open-mouthed (and you may well find yourself boggling at the amount of New World wealth they represent, as well).

Walking through its galleries is akin to the first time you read Shakespeare: "He coined that, too?" becomes "They have that, as well?" Fantastic treasures from every epoch and of every sort greet you around every corner – and there are a lot of corners. I'm in New York pretty often; I visit the Met regularly, and on each visit I still find rooms I haven't discovered before, and treasures I wouldn't have dreamed could still exist. My absolute favourite, one of those personal totem-objects that walking around museums sometimes gifts into your life, is a sea-shell – a perfect, hand-size sea-shell, spiny as a bird's foot, carved millennia ago by some unknown genius, from marble that sparkles like snow. The rooms, the spaces in the Met, are good and pleasant places to be in too (this not always the case in New York). And who's the woman over there, who looks so much like Susan Sarandon? Oh. It is her.

But for truly breathtaking surroundings, after the Met, can I recommend a trip uptown? The Met has a satellite, The Cloisters, housed at the end of a very long subway ride, on a bluff overlooking the Hudson River. The building itself, constructed between 1935 and 1938, was designed to look like a medieval abbey (a huge medieval abbey), and within it are objects and paintings and manuscripts and fixtures and fittings and frescoes and windows and entire rooms and even an entire cloister salvaged from other, lost medieval buildings across Europe. On a crisp autumn day, or in the summer, when the garden of medieval herbs is flowering, you're in a world where neither Manhattan nor the 21st century exists.

For modern art, you need to go back to the Met, and Museum Mile. The Met has had some superb modern art exhibitions of its own in recent years, but this time, begin with the Guggenheim. The Guggenheim has a splendid programme of exhibitions – sometimes absurd, often very odd, regularly fascinating – and its own collection of works by the Impressionists and Post-Impressionsts is not exactly to be sniffed at, but the real glory of the place is the building, designed by Frank Lloyd Wright. From the outside it uncurls, from the inside it embraces you. Go see it, and see if you don't agree.

The other biggie in the modern art world is of course MoMA, in mid-town. I have seen some stunning shows at MoMA – Van Gogh and the Colours of the Night was for me about as good as an exhibition can be – but the museum itself? There's something about the coffin-like lowness and funneling purpose of the entrance hall always makes me feel I'm about to be force-fed art, like a sort of cultural Toulouse goose.

Museum Mile stretches up the east side of Central Park. If you walk over the park, and yes, you can, perfectly and entirely safely these days (not forgetting to people-watch, and admire the show, obviously) you find the last of the Big Five New York museums – the American Museum of Natural History. Yes, it does look as astonishing as it does in Night at the Museum. Yes, it does have incredible fossilized skeletons; fantastic dioramas; a jaw-dropping collection of gems and minerals; a hall of meteorites; and a life-size model of a blue whale. If you first discovered museums as a kid, as I did, finish your own museum-postcard of New York by visiting this one, and rediscover the inner kid in you again.