Something could be causing a lean condition which does cause that to happen. ( such as: running out of gas, dirt in carb, clogged screen in tank, pinched fuel line, you get the idea)
Also be sure the governor arm is secure to the carb and not bent or rubbing anything.
Be sure the butterfly is closing and not binding when the cable is moved to idle position. The cable can sometimes slip off and not push it closed
There are not separate circuits for idle and high speed