My opinion on the desirable icon style is here:
Other than that, it's difficult to give advice. All I can say is that e.g. the group/ungroup and z-order icons are ugly and noisy, while the flip icons are OK. Icons IMHO must be simple and laconic, not "realistic" or "3D". Just go on and try to draw something and we'll comment :)
As for the procedure, draw new icons in share/icons/icons.svg, group each icon and assign an id to the group via the "item properties" dialog (you can also use the XML editor), and then you specify that id for the corresponding verb in verbs.cpp (the last component of each line of props).
I'd like everyone who commits icons.svg to also make sure they have the following in ~/.config/inkscape/preferences.xml, before they save the file:
<group id="svgoutput" usenamedcolors="0" numericprecision="7" minimumexponent="-8" inlineattrs="1" indent="0" />
This makes quite a difference in file size, and for such time-critical file as this, every millisecond counts. Thanks!