Joel Fitzgibbon's decision to quit the front bench has riled some