To start, I am an animal lover, I have 2 dogs, (one's a dog I brought home from a park area where I was photographing birds, and we also have a cat that lives inside.
I have a neighbor that always has a cat, one that is not spayed or neutered, I might add, and they have always come to my yard to 'hunt'. One of their cats had babies under my garden shed about 4 years ago, she had several kittens, 6, I think... They were really cute, and I could do nothing about it, but let them grow up, I wasn't about to take them to possibly be put to death by the City Pound. My neighbors are the worst, we are not on very good speaking terms, expect the mother, she is very nice, but does not understand how things work in America, she doesn't know what "No!" means with her ADULT kids, (They live at home still) so it's a waste of time to try and explain that cats should be neutered.
Anyway, the kittens grew up, and in the past, this has happened with other cats claiming our yard as their hunting grounds, and 'home', and every one died of natural causes. (If you include climbing underneath my truck and my wife's car in the Winter, then we turn on the ignition, not knowing) They also froze to death in the past, and other things that happens to feral types. I assumed this would happen again, but for some reason it didn't, immediately a couple of the cats got pregnant, by the uncle, no less, so I knew that nature wasn't going to take care of this, I had to do something, I should have already, but, that's what I get for 'assuming'. We had 11 cats now, they were using our yard as their 'catbox', play area, fight arena, bird eatery, you name it, they OWNED our yard. (They killed frogs in my ponds just to play with them until they died, that really upset me) Of course, I wasn't happy, but my wife is way more tolerant than I am, so eventually I won out, and I made the decision to trap them. (I wasn't about to call the City, they told me once when I called about a neighbor that had trapped a possum and left it in the cage that they would euthanize. So this would be no different. He promised me that he would take the possum to the woods and release him, but...) Anyway, I had had enough, I placed traps and it took me forever to get them to trust the traps, (I bought 3 traps at Harbor Freight) finally, 'some' were trapped, 3 were never trapped. To say this was a nightmare, that is an understatement, I had to take each cat I trapped to the vet and pay to get them neutered and spayed. It cost us a few hundred dollars, not counting the traps, I don't really remember the total amount we have spent. Having them spayed and neutered means they are protected from being picked up from the City Pound as ferals, they clip an ear to 'mark' them. I was told the cats would not be as wild, and they would calm down soon, no fighting, not trying to chase birds, etc. That wasn't the case, I finally got sick of it and trapped them again, then took them to a place at a pond, near the woods. I know, I know, but, I was sick of them, and what I was considering in my mind was a lot worse than what I ended up doing. (I don't think I would've gone through with my thoughts, but I was desperate) Now, we STILL have 2 cats here from that litter, 2 sisters, they are not really bad, they are only being cats, but I wish they were gone, of course, that is an old argument between my wife and myself... Actually, there are actually 3 cats, the mother of these cats is still around, she has been ousted by her daughters though, they don't allow her anywhere near, if they can help it, so we feed her too. (I actually don't mind her, she has never to my knowledge killed one bird here)
So, after that horribly long story, if you even read it, it's best to nip it in the bud as soon as possible, don't take ANY chances, do what you have to do, it is not guaranteed to end well.
Now, if I see a stray cat anywhere in my yard, I pop it in the butt with my BB gun, yeah, I know, it sounds cold-blooded, but wait until you go through what I did, you'll get you one too.
There are those of you that are thinking why I didn't go to my neighbors and ask them to pay, they do not listen to reason, but there is a language barrier, plus, as far as I know, nobody works there, so I did what I thought best. I decide to eat the costs because I didn't feel it was worth the effort. I had attempted to go through (their) proper channels once before over their barking dog, that didn't end well either. They are the worst neighbors I have ever had in my life, well, aside from the ones that live on the other side of me, now THEY are really bad, no English, no attempts at all to try the least little bit to 'talk' about things. I know this sounds like a rant, and this last part is, I just live as close to Hell as you can without burning up, I think. 
My main point is, do something now, I am pretty sure that your cat situation is nowhere near as bad as mine, but still, a nightmare can erupt from just one cat, don't let it ruin your life! 
Really, I apologize for going on so long, but even though it was slightly cathartic writing this, I did it to possibly help out in the future, that's all.
I forgot to mention that the main reason we kept the (very young, just weaned) kittens at first was their mother was killed early on, so they had to be fed or die a slow death possibly, that is why we initially took on the extra responsibilities.
Randy