The Malaysian Maritime Enforcement Agency launched a search for the group after one Rohingya migrant found on the islet off the northern resort island of Langkawi told investigators that at least 24 other Rohingya were missing.
Following further interrogation, the agency found 26 Rohingya refugees hiding on the islet on Sunday, said senior agency official Zawawi Abdullah.
The group included 12 men, 10 women and four children, officials said.
The Rohingya were believed to have been transported by local fishermen from a main vessel further out at sea and let off near Langkawi so they could sneak in undetected.
The Rohingya have been detained for further investigation and will be handed to the immigration department, he said in a statement.
Two other Rohingya migrants were also arrested for suspected trafficking.
Hundreds of thousands of Rohingya Muslims have fled Buddhist-majority Burma due to a military crackdown, and many live in densely crowded refugee camps in Bangladesh.
Muslim-dominated Malaysia has been a common destination of boats arranged by traffickers who promise the refugees a better life abroad.
Last month, Malaysian authorities detained 269 Rohingya and found a body on board a suspected smuggling boat off Langkawi that they said had been intentionally damaged so it could not be turned back to sea.
The Malaysian Maritime Enforcement Agency later said that the 269 were transferred from a larger vessel on which dozens of people were believed to have died and their bodies thrown into the sea.
Muhyiddin Yassin, the prime minister, said Malaysia cannot accept any more Rohingya refugees as it struggles with an economic slowdown caused by the coronavirus pandemic.
But he said those already in the country will not be sent back to sea.
Malaysia is home to over two million illegal immigrants as well as some 180,000 refugees and asylum seekers, including 101,000 Rohingya, who are registered with the UN refugee agency.
