French language Frustration

A part time resident welcomes me.
I speak in French.
He says I can speak in English.
Then tells me I need to improve my French!!!!!!!!
I tell him I'm trying EVERYDAY to speak the beautiful language!

He also enquired if I lived here all the time. 
He told me how they normally always come for the whole of  JULY AUGUST and sometimes Easter time but that they came here during the winter on their return from ski-ing....... and there was no one -NO ONE for a whole week.....  He Agreed it is dire in the winter.  I told him I didn't have a choice not owning property elsewhere.

He was kind.
 
An hour later, it again gets to me and I feel depressed.

These things matter to me:
1. BELONGING... I have to belong somewhere.
2. Need security, safety, comfort of my own home - shared or not shared.
3. Social & Cultural meaningful discussion and contact with others.
4. Support and inter exchange with others.
5. Family
6. Financial Independence.

So if, I need 3 and 4, and fail with language skills,  there is only one thing I ought to do at this age of my life.
I should no longer try to live in a Fort Knox / French "chateau " / "winter prison"  which requires a family and renovation.