I am trying to analyse nicotine (pKa= 6,1 and 10,9)and its metabolite cotinine by pair ion chromatography. The conditions are the following: Hichrome 5C18, 15cm*4,6mm, mobile phase: 7% methanol, 2mM Na2HPO4, H3PO4 0.2%(v/v), 1mM octanesulfonate sodium (giving a pH=2). Unfortunately, or no peaks were detected, either bad peaks with big tailing were detected. What happens?

With ion-pair chromatography, you need to equilibrate the column until the surface is covered with the ion-pair reagent. This may take a few hundred milliliters. Other possibilities as well, but this is the most common one....

Although I have never used the column that you quote, I suspect that it is not one especially suited to the analysis of moderate/strong bases like nicotine. There are a number of these new types of column available from different manufacturers which are based on very pure silicas and give much reduced tailing for bases. It may not be necessary to include the ion -pair reagent
with these columns. You can check the peak shapes for nicotine on a number of columns in my papers
J.Chromatogr. A 844 (1999) 23, 769 (1997) 169, 738 (1996) 169, 636 (1993) 213. Of course, you will have to optimise the conditions for the separation you wish to perform. In the 1993 paper, there are some separations shown of nictone from other tobacco alkaloids but I did not study any metabolites.

I need a capillary column for cotinine analysis in urine of passive smokers .

Here is the link for nicotine/cotinine metabolites analysis without ion-pairing reagent.
Short column (3.0x50 mm), fast analysis (4 minutes), several peaks with base line resolution.

http://allsep.com/makeChr.php?chr=Chr_064

Please check it out as alternative to your approach.