Mooning at Michael Jackson, turning up to every opening and celeb party in town and then alienating all your new fans with a dark, depressed follow-up to your million-selling breakthrough album: is that any way for a pop star to behave? Well, he doesn't think so, either. Ladies and gentlemen, Jarvis Cocker: he used to be famous, but he's all right now.

He wants to meet in a cafe, somewhere green. He decides on Holland Park, even though he lives in Hoxton and from East to West London is a trek in the summer heat. The record company sends a car to pick him up, but he cancels it; he would rather take the Tube. On his way to the park, he goes shopping for some roll-on insect repellent and a camera, and momentarily forgets he has an interview to do.

The cafe is full of posh Kensington mothers pushing their toddlers around in Land Rover prams. In the distance, a skinny man, 6ft 4in tall, comes into view. Knowing he is a little late, he strides purposefully across the grass. Narrow brown trousers, tiny holes around the knee, the hem falling down. A brown shirt, revealing wisps of chest hair. Dark brown Birkenstocks; long, unkempt toenails. The biggest tinted glasses hiding a delicate face; slightly pursed, pouty lips and shy eyes. His accessories of choice: a thick overcoat and a small Dixons carrier bag.

The only man in pop not to own a mobile phone walks into the cafe and two teenage lads shout out his name as though paid to announce his arrival. "It's Jarvis Cocker!" Jarvis Cocker smiles at them, a brief, awkward smile that serves only to acknowledge them. The man who has been described as "Pop's Mr Sex" apologises for being late, asks for a bottle of water and searches for a packet of Silk Cut Extra Mild in his bag. He no longer wants to do the interview in the cafe; it's too crowded and too hot. He leads the way to a park bench, sits down, lights a cigarette and explains that when he left Hoxton it was cloudy, hence the overcoat.

The last time we met, in Sweden back in February 1998, the wind was so cold it burnt your face. Jarvis wore nylon trousers, a shirt, a flimsy jacket and light shoes, which led him to shiver and skid all the way from the venue to the tour bus. "Yeah, I haven't quite got it right today either, have I?" he says in his soft Sheffield accent. He pulls a face. "It's not very good here because I can't make eye contact. I'm looking at a bush." He giggles. "Maybe we should move; I feel a bit too much like I'm in a spy film: 'Yes, you must meet him round ze corner.' Let's go and sit on the grass."

Under the shade of an ancient chestnut, Jarvis folds his legs underneath him like a giraffe. He is an urban type, brought up in Sheffield but transported to London and St Martin's School of Art, where he got a 2:1 in film-making. He doesn't like the middle classes, so the Kensington mums are anathema to him. He is as pale as paper, translucent almost, yet he chooses to meet in a park during a mini-heatwave. Jarvis Cocker, what is going on?

He stubs out his cigarette and smiles. "I've always lived in the city and never understood the countryside. For example, I don't know the name of the tree under which we're sitting. The fact that the countryside was alien to me felt wrong. On the last Pulp record, This Is Hardcore, I documented what it felt like to face up to the fact that success exists only in a very man-made, artificial environment. Earlier on this year, a blackbird built its nest in my garden and I was very excited about it. I even monitored the progress of the chicks hatching and learning to fly. It sounds a bit sad, but I guess I was going back to basics. And I bought this insect repellent today because I'm going on a trip to the Arctic Circle. I'm going back to nature."

A dog yaps insistently a few hundred yards away. Jarvis scowls. "If that fucking dog could shut up. It's doing my fucking head in. I can't concentrate with all that noise." His mouth curves upwards, hinting at a smile. "That's a bit of nature which could do with taming."

Three years ago, around the time of This Is Hardcore, Jarvis was busy making public the fact that, after years of waiting for it, he didn't actually want to be famous. He started Arabica Pulp (they soon realised the mistake) in an economics lesson in 1978, simply because he wanted to entertain. He also, presumably, saw pop stardom as a way out of what had been a painful childhood. A father who emigrated to Australia when he was seven and left him in a houseful of women; a mother who sent him to school in lederhosen; a pale, lanky body that made him feel physically awkward and unattractive. Pulp offered Jarvis glamour. It allowed him to be a misfit who was accepted rather than rejected, whose nerdiness could be appealing. He could have pursued academia (he failed an Oxbridge interview by pretending to have read a book he hadn't and deferred going to Liverpool University so many times that his place was withdrawn) but he was more interested in music, film and art.

Like many teenagers, Jarvis relied on John Peel's Radio 1 show to inform him about new bands. In 1981, aged 17, he went to see Peel at a roadshow in Sheffield and gave him a tape of Pulp's music; the DJ responded by offering the band one of his celebrated sessions, a milestone for any alternative band. Although Jarvis had created the band as a concept in that economics lesson, this was the real birth of Pulp.

In 1983, they released their debut album, It. Jarvis had never had a proper girlfriend - he didn't lose his virginity until he was 19, as documented on Pulp's 1994 single "Do You Remember The First Time?" - but this didn't stop him writing songs about girls and relationships. Four years later, the band released their second album, Freaks. The inspiration for the title was simple: living in a warehouse populated with misfits, Jarvis was worried that he was becoming a freak. Throughout this period, Jarvis was signing on and undergoing a series of minor disasters: in 1985, he fell 30ft from a window while trying to impress a girl, broke his pelvis and spent two months in a wheelchair. Eventually, in 1988, he moved to London to attend St Martin's and to prepare himself for full-time pop stardom.

Only it took much longer than anticipated. The real turning point for Pulp was Glastonbury 1995, the summer they filled the Stone Roses' Saturday night headline slot at the last minute and moved from cult status to mainstream in one evening. The band's performance was breathtaking, but there was one song in particular that struck a chord with the audience. "Common People", Jarvis's story of a working-class Northern lad meeting a posh girl at St Martin's, made Pulp the people's pop band.

So, this year, at 37 years of age, Jarvis is celebrating 20 years of Pulp. That he is still occupying his position as our most eccentric pop star - the US has Michael Stipe and Moby; we have Jarvis - is something of a miracle. Following that balmy night at Glastonbury, he and his band immediately began to attract tabloid attention. On 20 September 1995, The Mirror responded to the release of Pulp's single "Sorted For E's & Wizz" with a hysterical cover story: "BAN THIS SICK STUNT: Chart stars sell CD with DIY kids' drugs guide". Middle England was appalled by the explicit drug-taking diagrams on the single's artwork. Jarvis was delighted.

Less than six months later, Jarvis made the tabloid covers again. In February 1996, at the Brit Awards, he drank too much and decided to climb onto the stage during Michael Jackson's performance. He ambled around on stage for a few seconds before bending over and pretending to break wind. Security guards dragged him off the stage, he was taken to the local police station to sober up and the next day, The Mirror announced "Jacko Pulps Lout Cocker".

Jarvis was now a bona fide star. Pulp were in the charts and their singer's every move was under scrutiny. If there was an opening - an art gallery, a new shop, anything - Jarvis, would be there. And more often than not he would be pissed. The lyricist who wrote urban kitchen-sink epics and seedy, Blue Velvet-style love songs, the entertainer who modelled himself on Joe Orton and Alan Bennett, was fast becoming a self-parody.

Jarvis reaches for another Silk Cut Extra Mild. How does he feel now when he looks back on that period? He sighs. "I don't dislike myself for the way I behaved. I just feel a bit sorry for myself I look at things and say, 'God, you sad get'. And sometimes I look back and I am ashamed or embarrassed. I think I was just a bit lost, really. I suddenly had access to women I would never otherwise have had access to and it excited me. I had access to the finest-quality fanny available. But somehow it didn't seem to be providing the satisfaction I had thought it would. You know, I'd started the group when I was 17 and I had this dream... and when that dream became reality, I thought that I had to suddenly live my life differently, to go to openings and hang around with a different set of people."

He pauses, picking at the label on the bottle of water. "It's a bit pathetic, I know. It's like that Woody Allen film, The Purple Rose of Cairo: you're watching this film and suddenly you become part of the action. But the action didn't suit me. I would compare it to a nut allergy. Some people have just one peanut and their whole head swells up and they might even die. Well, fame didn't agree with me at all. I'm not sure why, really; it just didn't. And what made it worse was that one part of my brain kept insisting, 'You've worked towards this for so long, it's what you've always wanted.' But I wasn't right happy with it. Which made me feel like an idiot."

Jarvis wants to get it right so badly, to be fashionable in an anti-fashion kind of way, to lead and not to follow. Yet you get the sense that he is not very happy with himself, not particularly at ease with himself. Has he even learnt to like himself? He smiles. "Oh no. I generally don't like myself actually. Not really." Has he tried therapy? He laughs. "No, I'm from the North. People in the North frown upon that kind of thing."

Jarvis reached the depths of hopelessness in 1998, around the time of This Is Hardcore. A dark, bitter album, it reflected his negative response to the fame game. "It's very hard to write songs about being in that situation. The only song I've heard in recent years to effectively address the madness of celebrity is Eminem's "Stan". I thought it was very clever, using the fan letter as a premise. It's the thing that does your head in most when you're in a band - people knowing more about you than you do yourself. Coming up to you and talking about a cheese sandwich you ate at the motorway services outside Preston three years ago."

Although he recognises the brilliance of "Stan", Jarvis wrote a pretty good song about the pressures of fame himself. The title track of This Is Hardcore is a violent reaction to his experiences of life as a celebrity. "Yeah, I would agree, it is good. It frightens me. It does, it really does, because it's from a very scary place. It's like a snapshot of a mental landscape that I wouldn't care to revisit. But I am proud at bringing something back from that place. That song has a certain sleaziness and a certain atmosphere of decadence and decay. Whenever I hear it, I'm reminded of the dreadfulness of that time."

When he talks about this period of his life, Jarvis makes it sound as though he may have had a breakdown of some sort. He deftly avoids giving a straight response to such a suggestion, however, choosing to express it in the form of a metaphor instead: "It's like people eating kebabs. After you've had a really messy night out, sometimes the only way round it is to have a kebab. You're eating it and you know it's shit and it's probably been stuck on the stick for a fortnight and you'll get gastroenteritis the next day... but there's a perverse pleasure in it."

Over the past few years, Jarvis Cocker has done a lot of thinking. He has probably re-evaluated his entire life. The new album reflects a new-found optimism, replacing the paranoia of This Is Hardcore with an almost childlike innocence. Produced by reclusive legend Scott Walker, who lends it a flamboyant touch with strings and a choir, the album embraces nature in some classic Jarvis lyrics ("The trees/Those useless trees/Produce the air that I am breathing... They never said that you were leaving" on new single "Trees").

Jarvis was initially nervous about working with Walker, a childhood hero. "He's never produced a record for anyone else before, but we had appeared at the Meltdown festival he curated a few years ago and when our manager approached him, he seemed interested. He really understood what the atmosphere of each song should be." Jarvis empties the last of the mineral water into his mouth. He smiles. "The daftest thing happened. On the album there's a song called "Bad Cover Version" which includes a list of substandard records, one of which is the second side of Scott Walker's fifth solo album, Till the Band Comes In. I kept on thinking that I had to say something to him about it, but I just couldn't bring myself to. So one morning, I went into the studio thinking, 'I can't pussyfoot around any longer. I'll go straight and tell him.' He must have thought I was a nutter. He seemed bemused by it all."

Although Jarvis is excited about his new career as a low-key pop star, he no longer wants to rely entirely on music. Around the time of This Is Hardcore, he appeared on Channel 4 presenting Outsider Art, a series about artists working outside the establishment. He dismisses his performance as "pretty rubbish", despite the fact that he suited the subject perfectly. "I didn't want it to be a vanity project. I didn't want to spoil it by being a knobhead. My social skills at the time were so poor that really I would rather have just made the programme and not been in front of the camera."

He is now hoping to direct a feature film of Harland Miller's book, "Slow Down Arthur, Stick to Thirty". The rights have been sold to DNA, the company that made Trainspotting, and Jarvis thinks his eye for detail and love of social-realist film-making may help his cause. In preparation, he has been watching his usual selection of obscure world cinema (Alejandro Jodorowksy's El Topo, anyone?) alongside the recent crop of British films. "Most of which have appalled me," he says with a scowl. "Although I have to admit being surprised by Billy Elliot. Before I went, I thought they were bound to get it all wrong. But, amazingly, it worked. I almost cried. I was pretty near. I had a lump in my throat."

In between making the new album and trying to become a film director, Jarvis has also been DJing with Pulp's bass player, Steve Mackey, at London Fashion Week shows (until recently, he was going out with stylist Camille Bidault-Waddington) and at small, underground clubs around the capital. He does it simply because he still loves music. "And clubs are useful because people dance and get off with each other. As a DJ, you're providing a context for that to happen." He smiles. "If you get too drunk at a party and you are no longer capable of making coherent conversation with anyone, then as long as you can pick up the needle, you can always DJ."

Nature boy stretches his impossibly long legs out in front of him and watches a squirrel dart up a tree. "I have to say that I do find nature quite... awe-inspiring. Sometimes I think it's much more interesting than art. Both music and art are produced by human beings, who I understand a bit - I think I find nature impressive because I don't understand it."

These days Jarvis says he's much happier cycling round London than going to some shop-opening. He likes the fact that he has lived in the city for 13 years but can still find himself cycling down a street for the first time. He likes the idea of things happening by accident because he is more relaxed about how he leads his life now. "During the height of the fame thing, the random element seemed to disappear from my life and I didn't like it. I didn't like it at all."

The new Jarvis, apparently satiated with fame, is even considering moving back to his home town. "I never intended to stay in London this long. I do love it here, but there are so many distractions. Day-to-day existence is the most interesting aspect of life, but that gets forgotten about in London. In Sheffield, it's the most important part of everyone's life." He picks up his cigarette stubs and dead matches, drops them carefully into the empty water bottle and smiles. "And there's some very nice countryside round there, too."