Could a Brown win hurt the GOP?