Hi,
The number of columns and the number of rows you select define the total number of elements on a page.
This has to be fixed so that the MySQL query loading the product data from the database only loads the products necessary for the current page.
So when you select 4 columns and 2 rows, the system has to load 8 products on each page.
However, if you configure your listing to group the products by category, then on each page of the pagination, the products will be grouped by category. In that case, the products will still be displayed in the same number of columns.
And regarding the number of rows, it actually depends on the number of categories.
If each product on the current page is in a different category, then you'll end up with 8 rows as you'll have 8 groups of products.
That's how it is supposed to be.
So as Philippe said, if you don't want that, you don't want to group your products by category.
Now, regarding that issue with the "all sub elements" option, I'm not sure why you would have that. Turn on the error reporting and debug mode settings of the Joomla configuration and try again. You should then see an error message at the bottom of that page with more information as to why this is happening. Please provide the error message and we will be able to better understand the situation.