My opinion on the desirable icon style is here:
Other than that, it's difficult to give advice. All I can say is that e.g. the group/ungroup and z-order icons are ugly and noisy, while the flip icons are OK. Icons IMHO must be simple and laconic, not "realistic" or "3D". Just go on and try to draw something and we'll comment :)
As for the procedure, I think you draw new icons in share/icons/icons.svg, group each icon and assign an id to the group via the "item properties" dialog (you can also use the XML editor), and then you specify that id for the corresponding verb in verbs.cpp (the last component of each line of props). But I never tried that so I may be wrong. Try it and document what you find out.