There are at least two possibilities.
separate UID and advanced routing
If you can run this tool under a dedicated UID (i.e. nothing else uses it) then you can use
iptables to mark the packets for this UID (
--set-mark). This mark can be used in
ip rule for selecting a separate routing table.
separate namespace and advanced routing
You can set up a separate network namespace (
man ip-netns). You can connect it to your regular namespace with a veth pair. Then you set up routing rules for packets from this veth and run the tool within this namespace:
ip netns exec toolns tool