I understood Ilsa's love for her husband as that of a young, small-town girl for a wordly hero. Married to him she feels important, useful. But that's not romance. As for Victor, he loves her, but he loves his mission more.
Rick was the first adult love Ilsa had. They loved each other for each other - the world be damned. I think if she weren't married (with the sense of duty and obligation that carried) but was instead only engaged to Victor, she would have never left Rick in Paris - or in Casablanca.