The trick is to define inner sep=0pt when beginning the tikzpicture environment, as I learned from pgf bug report #172 minimum size in regular polygon fails for small length from last year. Here is a minimum working example, adapted to my use case of inline polygon symbols with roughly text height, i.e. 2ex in my case:
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{shapes}
\newcommand{\poly}[2]{\tikz[inner sep=0pt] {%
\node[regular polygon,%
regular polygon sides=#1,%
regular polygon rotate=#2,%
minimum size=2ex, fill] (0,0) {};}}
\begin{document}
Triangle upwards \poly{3}{0},
triangle downward \poly{3}{180}
and diamond \poly{4}{45}.
\end{document}
This snippet yields the following result:
