Try leaving a glass of tap water on the table for a day without touching it. You’ll notice the sides of the glass are speckled with bubbles. Where did this gas come from? It came right out of the water, in fact. As the water passed through the pipes on its way to your home, gases from the surrounding air were dissolved into it.
As the water warms, it is able to hold less gas. Boiling the water removes a lot of the dissolved gases in the water, just as a glass of water left out on a table will release bubbles.
Freezing the water does the same thing. Ice is formed as the water molecules line up with one another, which also pushes out dissolved gases and minerals. Because the water freezes from the top down, the dissolved gases are pushed slowly to the bottom until it collects into bubbles. Filtered water has less dissolved minerals in it, also making for clearer ice cubes. So should you be using non-filtered water for this activity, to make the difference more obvious?