Gaviria looked as though he had gone too soon as Greipel charged down the middle but the Colombian came back strong with a second jump to win the stage, just holding off a late charge from Sagan.
He now heads team-mate Tejay van Garderen in the general classification, with Team Sky's Geraint Thomas three seconds back in third and defending champion Chris Froome a further 52 seconds behind in 17th.
Sagan was closing fast and seemed to be on pace to overtake him just when Gaviria hit the line.
"He is faster than me", Sagan said about Gaviria. And maybe we will see the next days on the climbs.
Nairo Quintana would have been in the winners' category had it not been for the first stage but as it was he now has quite a defect and this is before the cobbles and he's a minute down on every one of his rivals.
Riders were enjoying a calm sunny afternoon in the cycling hotbed of northwest Brittany until a pileup near the front of the peloton with just over 5K left that sent several riders to the tarmac. Then with 5km to go a big crash midway down the peloton threw another spanner in the works.
But there were no changes among the title hopefuls.
Tom Dumoulin was seventh (11 seconds behind), last year's runner-up Rigoberto Uran was 10th (:35) and Porte 14th (:51).
Movistar will be delighted that Mikel Landa and Alejandro Valverde are right on the heel of the Sky train, with Valverde having plenty of cobbled form.
Behind them, the pack was led by Quick-Step Floors and BMC and the day was quite uneventful, even though there was a small crash at around 30 kilometres to go that affected riders like Jakob Fuglsang (Astana), Robert Gesink (LottoNL-Jumbo) and Tony Gallopin (AG2R La Mondiale) but didn't leave any consequences.
Wednesday's Stage 5 is a hilly 204.5-kilometer leg from Lorient to Quimper.
His absence means Bardet's team is now down to seven riders with 17 more stages to be raced before the Tour's end on July 28.
Froome, who was cleared of doping allegations last week by the International Cycling Union, has been jeered by some skeptical fans since arriving in France.
The Manxman is aiming to break Eddy Merckx's record of 34 stage wins but remains four off that mark.
- Barcelona sign Lenglet from Sevilla for £32m
- Historic towers at Cape Canaveral launch pad demolished
- Brooklyn Nets trade Jeremy Lin to Atlanta Hawks
- Iranian Supreme Leader's Adviser Hails 'Very Constructive' Meeting With Putin
- 'First blue whale' caught in 50 years
- Arsene Wenger explains why Paul Pogba has improved at the World Cup
- Industrial park blast kills at least 19 in China
- Putin sees Trump as "counterpart" at upcoming talks
- Malankara Orthodox Church priest held on rape charge
- Obama loses millions of followers in Twitter purge