I have a 1988 V6 2-door Toyota 4Runner with 120k miles. When the temp is below 50 degrees, the engine idles erratically, but only when the brake is depressed. The normal idle drops from about 1100 rpm to about 700-800 rpm, and then immediately returns to 1100rpm when you take your foot off the brake.
It sounds like you have a bad power brake booster. The next time you start it up put your head down under the dash on the driver's side while depressing the brake pedal, if you hear a notable "hissing" sound this would confirm a brake booster failure.
The booster uses engine vacuum to assist in brake application by means of a rubber diaphragm applying pressure on the back of the master cylinder. After time and mileage accumulate, this diaphragm can become brittle and crack, causing an excessive loss of vacuum while applying the brakes.
The hissing noise you're hearing is the leak that's causing the erratic idle which you've described. The reason a vacuum leak has more of an effect on idle quality in cooler temperatures is that a cold engine requires more fuel (a richer mixture) in order to run properly. A vacuum leak adds more air to the air/fuel mixture causing a lean condition.
Your best bet is to replace the brake booster with a new one, but that can get expensive, so you may want to find a good used one and save some money.