How do I know if it's time to leave my husband?

you should leave him and just think if you leave him now you both will most likely remain good friend rather than finding out he has a guy behind your back which would put that anger between yous. therefor you can find a decent respecting of you kind of guy. it'll be healthier for the kids too if yous separate on good terms. good luck