Restrict remaining product options??

If you want to use characteristics there is no option to hide the dropdown in such cases.
What you can do is have a value "no color" for the color characteristic and in your products' manage variants button, you can disable al the variants with the cloth fabric and the color being something else than "no color".
Then, on the product page, instead of having the add to cart button, they will get a "no stock" message that you can change with a translation override to something like "no color possible for cloth, please select the "no color" value", or something like that.
Or you could change the "characteristic display method" option of the configuration to "list" in order to display a table of all the available variants of the product with an add to cart button for each variant.

An alternative is not to use characteristics but custom item fields which can have their display restricted to the value of another custom item field with the "display limited to" option of the custom field. But custom fields won't allow you to manage the information of the product per variant (and thus you can have different quantities of the product for different variants for example).