Will my song drop out of the chart if it has few votes?

No. A submitted song stays in the chart permanently, no matter what position it's in. Only the votes are reset every week.