Alice, this has been my experience of them as well. I have one in my garden, placed in a little sunny hollow behind a building, in a patch of half-decent soil (rare in my garden) about 200m from the sea but protected from the wind. I bought it in the autumn of 2010, and upon planting, it rapidly lost its leaves and did not get them back for many months (I did wonder whether it had given up, but the trunks remained firm). It put on healthy leaves last year, eventually, but no flowers. This year, the same pattern so far, the leaves have only just come out in the last week, and it remains to be seen whether it gives a flower. It gets watered twice a week, though I plan to reduce this once it is established.
It seems to take a long time to make up its mind, but I am hoping the wait will be worth it. I have seen a couple in local front gardens that look utterly fabulous and happy - many-branched, full of big leaves - so time will tell if those are isolated examples (pumped full of water and fertilizer) or not. I am trying to imagine something that could clothe the lower branches while they are bare, but that would not detract from those lovely leaves once they appear. Perhaps a summer-deciduous clematis, which could be strategically pruned or woven into the branches at leaf-out, so it would remain hidden over summer, but give leaves, flowers and interest in the winter? C. napaulensis comes to mind. However I have just checked out the spec for this species and it gets rather large - 5m - so will perhaps get too big for its boots in a frangipani...