That seems a good idea to me.Perhaps we should have a 'long blue' to either return to default sort or return to previous sort. At the moment I like the 3-2-1 sort but occasionally do the next sort (AZ). However, then I want to return to 3-2-1 which is five presses of the 'short blue' button... A 'long blue' would be convenient and would not change other functions.
As it is now I never use the colour button, exactly because of the reason you stated + the fact that the actual way of sorting can hardly be distinguished; instead I always go into the context menu to change sorting.