There is no fence. The Rebels are an unorganized group of insurgents hellbent on disrespecting loyal Imperial citizenry and their property. Their sole purpose is to destroy it and cause anarchy. Their leader, when we find him, is a pariah. Disinterested in individual economic advancement and personal success. What you see is a wide gap of famine of bygone days of the old regime that left people to suffer, as the Empire races with a humanitarian effort to feed those who wish to climb to achievement.