It's not always that Schoenberg is out-weirded in a concert programme. More often, his is the eyebrow-raiser, the tougher music on display, which takes more effort to enjoy. But it's a sure sign of progress that a work like the string sextet Verlkärte Nacht (1899), whose première was apparently cancelled because of an overly dissonant inverted ninth chord, can sit so securely in a concert programme today. And actually, it was made to seem positively tame last night, placed next to Schubert's perpetually perplexing String Quintet (1828).

Janine Jansen's final concert in her Wigmore Hall residency saw her joined by a starry lineup of string players, including violist Maxim Rysanov and cellist Thorleif Thedéen. The group are in the process of touring this rich programme of music around Europe. They may lack a name, but the ensemble lacks little in terms of cohesion, and they played both works with a sense of care which suggested that the players' obvious individual excellence had been coupled with diligent preparation.

The group's sincerity of purpose was apparent from the off in Verklärte Nacht ('Transfigured Night'): they played the piece's gradual, soft opening incredibly delicately. It almost felt as if the relative inexperience of the group in playing together had heightened their awareness of the need to communicate. And the slight tension of their coordination was perfectly suited to the foreboding atmosphere this opening requires. They didn't disappoint as the piece developed either: the numerous climaxes were well controlled and the tricksy pizzicato sections played with preposterous facility. The denser contrapuntal sections felt less balanced, with a slightly decreased sense of whole, but actually this almost brought out the frenzied nature of Schoenberg's writing yet more.

It's a quirk of Verklärte Nacht that despite perceptible flaws in overall structure – the D major climax arrives so early that there is a little too much thumb-twiddling late on – it remains one of the most genuinely compelling listens in the chamber music repertoire. Such is the strange transparency with which Schoenberg interprets the poem by Richard Dehmel on which the piece is based. Jansen and her colleagues caught the work's inherent sense of drama perfectly, and the closing D major swell was an appropriately rich end to a deeply romantic piece.

The Schubert String Quintet in C is a considerable contrast. While the Schoenberg is a straightforward trajectory from darkness to light, from sin to redemption, the Schubert is haunted throughout by a perpetual darkness. Even the calmer moments – the slow movement, the trio, that odd passage in E flat in the first movement – seem somehow unbelievable, like dreams. The group played these sections with the same care and poise as in the previous piece, and the curious textures of the slow movement were a particular highlight, with the gentlest of plucked cello lines and the softest of playing from Jansen (who excelled in soft playing all evening). The nightmarish middle section of this movement, which shifts up a semitone to F minor, was just as violent as necessary, and the drift back to the uneasy tranquility of the opening was subtle and soft. It was again the denser moments which worked less well, and the Scherzo, a manic barn dance, sometimes felt a little too prettily phrased, though the players still caught its rusticness effectively.

The sharp rhythmic characterisation of the sprightly final movement was enjoyable, and the players' technical virtuosity was most appreciable in the closing stages, where Schubert ratchets things up to a manic Presto. The quintet ends with as bizarre a gesture as you will find in the classical canon – a forte, unison C preceded by a chromatic grace note on D flat – and if the previous fifty minutes of weird harmonic shifts and textures hadn't already made the point, this ending really makes it clear that Schubert's piece is several times as daring and unconventional as anything in Verklärte Nacht. The group played this ending with the confidence and ease of an established chamber group, which is more than could have been expected. I only hope they continue to perform together: all they really need now is a name. 'Transfigured Soloists' would get my vote.