Hours of heavy rains overnight caused rivers to burst their banks, flooding homes with mud in Putumayo province.
The Red Cross said that at least 220 were missing, and another 202 were injured.
President Juan Manuel Santos, who travelled to the area, said troops had been deployed as part of a national emergency response.
He declared a state of emergency in the region.
Reuters news agency said more than 1,100 soldiers and police officers had joined the rescue effort. One army officer said the main local hospital was struggling to cope.
The region’s governor, Sorrel Aroca, told Colombian media that whole neighbourhoods had been buried.
Rescue services said their efforts had been hampered by continuing bad weather and damaged infrastructure.
“There are mobility issues on almost 80% of the roads, and from where the road ends, it is three hours to where the landslide took place,” said one police officer.
Bridges have also been swept away.
Jose Antonio Castro, mayor of the provincial capital Mocoa, told Caracol radio that the town was “totally isolated”, without electricity and water.
On arriving at the scene, President Santos told reporters, “My heart and the hearts of all Colombians are with the victims of this tragedy.”
An alarm had sounded as the river’s levels rose, leading many people to leave their homes for shelter, and avoiding wider loss of life, Colombia’s El Tiempo newspaper said.