Did I read too much into my last doctors appointment?

Yesterday I had my post-radiation therapy follow up appointment. Everything went well - my site looks great and is healing well, so he released me from his care.

As I've recently noted, I'm in the position where I really would rather not do chemo - but since I was unable to get an Oncotype DX test done on my tumor, I honestly have no clue as to whether I would benefit from chemo therapy or not. my research show me that the Oncotype DX test is most effective for first time diagnosis, as it maps out your risk of recurrence and whether or not chemo would be beneficial for you or not. Since this was a recurrence for me that test would be useless. So I figured I'd ask my radiation oncologist his thoughts on it. you know how you'll talk to one doctor and they'll say or you don't need this treatment you only need this one (which happens to be the one they're pushing.)

OK, so I've mentioned how I'm not sure if I want to do chemo because of how it may affect my liver. This doctor assured me that there are tests they can do to monitor my liver function - which I know about. However these tests seemingly some how missed the fact that my mother had cirrhosis until it was already end stage, and was too late. Which still has me baffled. Thus the reasoning behind why I don't want anything that will be bad for my liver as I take plenty of medication and like an occasional drink. My mom NEVER drank, but was on all kinds of prescriptions - though they blame her cirrhosis on a gastric bypass procedure she had done in 1979 - a procedure I would never undergo now because of it's known liver complications.
So here I am looking for some kind of advice from my doctor to support my reasoning when he tells me something that I has me wondering did I hear him right or am I over-dramatizing the conversation. I asked him if he thought chemo would benefit me, and if this was the end. According to him the radiation eradicated the cancer in my breast, so it will likely NEVER return there (or at least SHOULDN'T) - BUT - since I have already had a recurrence is is possible that the cancer CAN return somewhere else. WHOA, did he just tell me that my cancer will likely return or did I make a mountain our of a molehill? I am still confused about what he told me (he has a very heavy accent, so it's hard for me to understand him at times.) He didn't really say yay or nay on the whole chemo or no chemo thing. It's just chemo is a systematic treatment and radiation is a localized treatment. Also what I know about this recurrence is that it is believed that 3 years ago when I was first diagnosed with breast cancer and had a mastectomy that one tiny little cell was left behind and that my medication Femara failed to prevent my estrogen from binding with the cancer cells, so it grew. My only problem with this theory is that my pathology on my mastectomy came back clean and clear and I've had a hysterectomy - so how would it be possible for one little cell to be left behind and grow when my estrogen level should be very low? Also I did not receive radiation the first time - had I did then a chest wall recurrence would not have happened.

So am I looking at fighting cancer for life? Will doing chemo really prevent it from returning? As it is, my cancer has not spread - so why would I need chemo. Can't I just do hormonal therapy (a new AI treatment like Femara) and get annual or bi-annual PET scans to spot out cancer clusters - then deal with them as found? Because seriously, if the cancer were to return a PET scan would be the only way for it to be detected besides blood marker tests.

I'm still unsure of the whole chemo thing. In fact I think I am more confused now as to what to do. If it were you, what would you do?