I don't know about it "haunting me", but it sort of pervades the entire series. It'll be easier to stop bitching about it now that it's a straight shot to the finish though.You have to let go the fact that they split the season already, it's been haunting you since before episode 1 haha.
I'm glad that they're finally getting around to doing stuff with Emilia. I found it odd how this arc started with a lot of her and it quickly drifted away from that, but it just goes to show that Subaru was avoiding it.