Man struggling to move home against opposition from lots of pointing fingers