Not really the smallest - probably something could be made with only 1 brick, but 4x4 is still cramped. This is experimental and started with Q which was how My Bricks appeared one time and I thought “That looks like a Q”. And the rest, as they say…
The real problem (isn't it always?) is consistency of style across all letters. The T has nothing of the blobby fun of the E.

Just Caps and Numbers (for now?)

TwoDotty is just that -2 dot sizes and it's a bit daft. I enjoy exploring limits - how small? How few points of information is just enough?

Really basic rule based 5x3 blocky font with some curves and diagonals to make it work. K, Q, X and Z least successful IMHO(!). Designed for a clock so almost pure monospaced, but the punctuation characters are narrower for a cleaner look.