It sounds like you are in quite a quandry. What makes you say that she is unfit? 17 seems pretty young to be coping with 2 kids. Have you spoken to her. Perhaps she will be willing to let the kids live with you for a bit until she finishes school. Or you could share custody. This could save her & the kids some trauma
Have you tried any local community sources? Perhaps she is too proud to get help or is just realising that she is still a kid and panicking a bit. I can see how this has a potentially concerning impact on the kids. there might be some kind of parenting courses or social workers that can help her & you.
Is there some kind of non abusive local community (like a housing estate)for young mums where a social worker looks in on them and helps with the parenting?
Is she still at school? If so what about them? You and your daugher could try talking to the school counsellor or the principal about this issue & see what you can all come up with.
Have you considered the possibility that if she spends a year at tbs she will be away from the kids & completely cut off so her parenting skills will never improve. She is also likely to really resent you for doing this is such a traumatic and confrontational way. Are the kids going to benefit from being completely cut off from mum for such a period of time?
What about your extended family? Can they assist here?
Given that she is 17 she can forcibly leave any boarding school in a year anyway & retrieve the kids, so really this seems like an expensive short term solution.
I would try to remember that you have 3 kids who are in real trouble here& they all need you
