A lot of songs at that time were extremely bland, squeaky-clean stuff.
As he drew the character, he decided that a squeaky-clean square (like a kitchen sponge) fits the concept.
In the spirit of self-regulation, banks would have a real incentive to keep their affairs squeaky-clean.
