Cokin filters are square and fit into a holder that is attached to the front of the lens. this sort of set up is essential for using graduated filters. Lee filters are very similar in set up, but are higher quality and cost a lot more than Cokin.
The other kind of filters you can get screw onto the front of the lens. they are not really suitable for graduated filters, and come in varying sizes depending on the filter thread of your lenses.
For landscape photography 0.3, 0.6 and 0.9 graduated neutral density filters ( or grey grads with cokin) are ideal. tobacco grads are a bit 80's, they have kind of gone out of fashion. But if you like it, that's what matters.
Straight neutral density filters are useful for slowing the shutter speed to increase motion blur eg waves and waterfalls - but IMO this effect has also been overdone of late. also, a circular polariser will give a similar effect as they reduce light by 1-2 stops anyway.
For setting up, I'd suggest getting the Cokin P series, with the relevant adapters so the mounting bracket will fit all the lenses you want to use for landscape stuff, some grey grads ( Cokin don't make true neutral density) and a tocacco grad if you have to, and then get a good multicoated circular polariser that fits the lens or lenses you wish to use - Hoya and B+W are names that are recommended regularly.
I'd also advise getting multicoated UV filters to keep the front elements of your lenses clean - hoods can provide physical protection, but they don't keep lenses clean like a good filter. there are a lot of arguments about this, about half of us use them and half don't. Those of us who do, don't post threads like ' I scratched my lens - what can I do now?'
Final bit of advice, always keep filters scrupulously clean. any dirt increases the risk of flare. And buy the best quality you can afford.