Cable TV systems are 'closed' which means no signal is supposed to escape into the air. This allows them to use frequencies assigned to licensed over the air services including life safety services such as police, fire, aviation, military, etc. The cable network also uses the same frequencies that OTA TV broadcasts on. So, you don't want to hook an antenna to a cable service signal, you'll cause harmful interference. You'll also find the cable and OTA TV signals will interfere with each other, leaving you with no cable or OTA, or at least significantly degraded.
You can disconnect the cables that feed the two sets from the existing cable splitter. In fact, if the internet and phone service are delivered by the same modem, and it's the only device that will remain active on the cable connection, you can disconnect the splitter from the cable service and use it for distributing your OTA signal. Just run the one and only cable connection directly to the internet/phone modem.
The end result needs to be, cable service fully and completely separate from the OTA system.
I would try a non-amplified install to begin with. Use a 2-way splitter if feeding two TVs. Splitters with more ports have more loss. (Why add extra loss?)
If you prove the need for an amplifier, it needs to handle strong signals. The best place for an amplifier is close to the antenna. Amplifiers are only capable of compensating for losses on the output side of the amplifier.
If you where going to feed four or more TVs, I'd suggest a Channel Master CM-3410 if you already own an appropriate passive splitter. Channel Master also offers the same amplifier with a built in splitters in 2, 4 & 8 output configurations. (CM-3412, 3414 & 3418)
Unused splitter ports should be terminated using
F type 75Ω terminator caps