Why does my ford ka sometimes not start?

It use to have a shake when I first started the car (stupidly I just ignored it) and it gradually got worse, it now decides it doesn't always want to start.
Its not the battery I had a brand new one put in last week and tires were also new then.
It is 11 years old, it seems to be worse if I have driven it alot the day before, but then also if I haven't driven it for a day.
When it doesn't start I normally have to keep playing with it and it starts, the car runs perfectly after that. All lights come on as normal when I turn key, and it does attempt to start.
Any ideas?

Fuel pump or relay.