Have you ever found yourself in a situation where you had a bunch of parameters to tune for a better result, but there were just too many to exhaustively search all possibilities?

I recently saw a video that talked about how to reduce the number of experiments in these situations, such that it formalizes what conclusions you can and cannot make from those experiments. Even better, the number of experiments can be a sliding scale to either be fewer experiments (faster), or more information from the results (higher quality).

We are going to explore a method for doing that in this post, but you should also give the video a watch before or after reading, as it talks about a different method than we will! https://www.youtube.com/watch?v=5oULEuOoRd0

Fractional Factorial Design Terminology

The term “fractional factorial” is in contrast to the term “full factorial”.

To have some solid context, let’s pretend that we want to figure out how to make the best brownies, and we have 3 options A, B, C. Option “A” may be “add an extra egg”, “B” may be “double the butter” and “C” may be “put nuts on top”, where 1 means do it, and -1 means don’t do it.

Three parameters with two choices each means we have 8 possibilities total:

	A	B	C
Test 0	-1	-1	-1
Test 1	-1	-1	1
Test 2	-1	1	-1
Test 3	-1	1	1
Test 4	1	-1	-1
Test 5	1	-1	1
Test 6	1	1	-1
Test 7	1	1	1

We’ve essentially counted in binary, doing all 8 tests for all 8 possibilities of the three binary parameters.

If you’ve noticed that the number of tests are 2^3, and not a factorial, you are correct. The term “factorial” refers to “factors” which is another name for a parameter. We have 3 factors in our experiment to make the best brownies.

Each factor has two possible values. In the literature, the number of values a parameter can take is called a level. So, our brownie experiment has 3 two level factors.

If we didn’t have enough pans to bake all 8 types of brownies, we may instead opt to do a smaller amount. We may only do 1/2 of the tests for instance, which means that we would be doing a fractional amount of the full factorial experiment. We’d be doing a fractional factorial experiment. We could even do 1/4 of the experiments. The fewer experiments we do, the less information we get.

Choosing which four experiments to do, and knowing what conclusions we can draw form the results is where the magic happens.

Fractional Factorial Design

Going back to our brownie recipe, let’s say that we want to do 4 bakes, instead of 8.

We start by making a full experiment table for A and B, but leave C blank

	A	B	C
Test 0	-1	-1
Test 1	-1	1
Test 2	1	-1
Test 3	1	1

We now have to decide on a formula for how to set C’s value, based on the value of A and B. This is called a generator and to test things out, lets set C to the same value as A, so our generator for C is “C = A”

	A	B	C=A
Test 0	-1	-1	-1
Test 1	-1	1	-1
Test 2	1	-1	1
Test 3	1	1	1

We have 4 experiments to do now, instead of 8, but it isn’t obvious what information we can get from the results. Luckily some algebra can help us sort that out. This algebra is a bit weird in that any letter squared becomes “I”, or identity. Our ultimate goal is to find the “Aliasing Structure” of our experiments, which is made up of “words”. Words are combinations of the parameter letters, as well as identity “I”. The aliasing structure lets you use algebra to find what testing parameters alias with other testing parameters. An alias means you can’t tell one from the other, with the limited number of tests you’ve done.

That is quite a lot of explaining, so lets actually do these things.

First up we have C = A. If we multiply both sides by C we get CC = AC. Any value multiplied by itself is identity, so we get I = AC as a result. Believe it or not, that is our full aliasing structure and we are done! We’ll have more complex aliasing structures later but we can keep it simple for now.

Now that we have our aliasing structure, if we want to know what something is aliased by, we just multiply both sides by our query to get the result. Here’s a table to show what I mean.

Alias Query	Result
I	I = AC
A	A = AAC = C
B	B = ABC
C	C = ACC = A
AB	AB = AABC = BC
AC	AC = AACC = I
BC	BC = ABCC = AB
ABC	ABC = AABCC = B

Looking at the table above, we see that B = ABC. This means that B aliases ABC. More plainly, this means that from our experiments, if the brownies were rated from 1 to 10 for tastyness, we wouldn’t be able to tell the difference in results between only doing option B (doubling the butter) and doing all three options: adding an egg, doubling the butter, and adding nuts on top.

Another thing we can see in the table is that A = C. That means that we cannot tell the difference between adding an egg, or adding nuts on top.

Lastly, I = AC means that we can’t tell the difference between doing nothing at all, compared to adding an egg and adding nuts on top.

If we care to be able to distinguish any of these things, we’d have to change our experiment design. Specifically, we would need to change the generator for C.

We can do that, and in fact there is a better choice for a generator for C. Let’s make it so that value of C is the value of A and B multiplied together. C = AB.

	A	B	C=AB
Test 0	-1	-1	1
Test 1	-1	1	-1
Test 2	1	-1	-1
Test 3	1	1	1

It doesn’t look like much has changed, but lets do the same analysis as before.

We can get our aliasing structure by starting with C=AB and multiplying both sides by C to get CC=ABC, which simplifies to I = ABC. Let’s find all of our aliases again.

Alias Query	Result
I	I = ABC
A	A = BC
B	B = AC
C	C = AB
AB	AB = C
AC	AC = B
BC	BC = A
ABC	ABC = I

Something interesting has happened with this design of experiments (D.O.E.). All of the single letters alias with double letters. This means that we can distinguish all primary effects from each other, but that primary effects are aliased (or get “confounded”) with secondary effects. In many situations, the primary effects are more prominent than secondary effects (adding butter or adding nuts is more impactful individually, than when combined). In these situations, you can assume that if there’s a big difference noticed, that it is the primary effect causing it. If there’s any doubt, you can also do more focused experiment(s) to make sure your assumptions are correct.

Resolution

So why is it that our first generator C=A wasn’t able to differentiate primary effects, while our second generator C=AB could? For such a simple example, it might be obvious when looking at C=A, but this has to do with something called “Resolution”, which is equal to the length of shortest word in the aliasing structure (not counting I).

With I=CA, that is two letters, so it is a resolution II D.O.E. Resolution II D.O.E.s have at least some primary effects aliased (or confounded) with other primary effects. Resolution II D.O.E.s are usually not very useful.

With I=ABC, that is three letters, so is a resolution III D.O.E. Resolution III D.O.E.s have primary effects aliased with secondary effects. These are more useful, as we explained above.

As you add more factors, you can get even higher resolution D.O.E.s. A resolution IV has primary effects aliased with tertiary effects, and secondary effects are aliased with each other. A resolution V has primary effects aliased with quaternary effects, and secondary effects are aliased with tertiary effects. If you are noticing the pattern that the aliasing effect classes add up to the resolution, you are correct, and that pattern holds for all resolutions.

Getting a higher resolution D.O.E. is usually better, so you want your generators to contain more letters. You have to be careful to make them unique enough though. If you had four factors A,B,C,D and generators C=AB, D=AB, you can see that you’ve introduced an alias C=D.

If you work out the aliasing structure, this problem becomes apparent that you’ve made a resolution II D.O.E. Aliasing structures always need 2^p words, where p is the number of parameters that have generators. In this case, p is 2, so you need 2^2 or 4 words in the aliasing structure.

The first word comes from C=AB, which can be rewritten as I=ABC. Actually that’s the first two words.

The second word comes from D=AB, which can be rewritten as I=ABD. That gives us three words:

I = ABC = ABD

To get the fourth word, we can see that both ABC and ABD are equal to each other. Remembering that when we multiply the same thing against itself we get identity, we can make our fourth word by multiplying them together to get something equal to identity for our aliasing structure.

ABC*ABD = AABBCD = CD.

So, our four words all together are:

I = ABC = ABD = CD

Since “CD” is the shortest word that is not identity, and is 2 letters, we can conclude that we have a resolution II D.O.E.

We can multiply the aliasing structure by C to get all the things that C is aliased with.

C = AB = ABCD = D

This shows we cannot tell the difference between C and D, because they are aliased, or confounded, and confirms that we have a resolution II D.O.E.

What’s With The Weird Algebra?

When we said C=AB, where A and B were either -1 or +1 in each column, it was pretty easy to just multiply the values together to get the result.

What probably made less sense is when working with the generators and aliasing structures, why a letter times itself would become identity. The reason for this is that the multiplication is a component wise vector product. If the original value was -1, then we get -1*-1 or +1. If the original value was +1, then we get 1*1 or +1. The result of a component wise vector product between a vector and itself will have a 1 in every component, which is identity.

Nothing too mysterious going on.

Larger Example: 8 Experiments, 5 Factors, 2 Levels

To help cement these ideas, we’ll do one more example with 8 experiment and 5 factors. That means we’ll only do 1/4 of the experiments we would if doing the full factorial, doing 8 experiments instead of 32.

8 experiments mean that the first three factors will be full factorial, and the last two parameters will need generators. We’ll use D = AB and E=BC as our generators.

We can make our experiment table to start.

	A	B	C	D=AB	E=BC
Test 0	-1	-1	-1	1	1
Test 1	-1	-1	1	1	-1
Test 2	-1	1	-1	-1	-1
Test 3	-1	1	1	-1	1
Test 4	1	-1	-1	-1	1
Test 5	1	-1	1	-1	-1
Test 6	1	1	-1	1	-1
Test 7	1	1	1	1	1

Let’s also make our aliasing structure. Since we are using generators on 2 parameters, that means p=2, and so we have 2^p = 2^2 = 4 words in our aliasing structure. The first 3 are easy, since they come from our generators directly.

D = AB : I = ABD
E = BC : I = BCE
so:
I = ABD = BCE

For the fourth word, we again know that ABD and BCE are equal since they are both equal to identity. We also know that multiplying anything by itself is identity, so we can multiply them together to get the fourth word:

ABD*BCE = ABBCDE = ACDE

Our full aliasing structure is then the below, which shows that we have a resolution III D.O.E, meaning primary effects are only aliased with secondary effects.

I = ABD = BCE = ACDE

Lastly, let’s make the full 32 entry table showing all aliases to understand the things we can, and cannot differentiate from our experiment results.

Alias Query	Result
I	I = ABD = BCE = ACDE
A	A = BD = ABCE = CDE
B	B = AD = CE = ABCDE
C	C = ABCD = BE = ADE
D	D = AB = BCDE = ACE
E	E = ABDE = BC = ACD
AB	AB = D = ACE = BCDE
AC	AC = BCD = ABE = DE
AD	AD = B = ABCDE = CE
AE	AE = BDE = ABC = CD
BC	BC = ACD = E = ABDE
BD	BD = A = CDE = ABCE
BE	BE = ADE = C = ABCD
CD	CD = ABC = BDE = AE
CE	CE = ABCDE = B = AD
DE	DE = ABE = BCD = AC
ABC	ABC = CD = AE = BDE
ABD	ABD = I = ACDE = BCE
ABE	ABE = DE = AC = BCD
ACD	ACD = BC = ABDE = E
ACE	ACE = BCDE = AB = D
ADE	ADE = BE = ABCD = C
BCD	BCD = AC = DE = ABE
BCE	BCE = ACDE = I = ABD
BDE	BDE = AE = CD = ABC
CDE	CDE = ABCE = BD = A
ABCD	ABCD = C = ADE = BE
ABCE	ABCE = CDE = A = BDE
ABDE	ABDE = E = ACD = BC
ACDE	ACDE = BCE = ABD = I
BCDE	BCDE = ACE = D = AB
ABCDE	ABCDE = CE = AD = B

Closing

A limitation you may have noticed in this article is that it only works with binary parameters (2 level factors). Part 2 of this blog post will show how to overcome that limitation using a different technique for fractional factorial experimental design.

Until then, there is a way to extend this method to factors which have a multiple of 4 levels. If you are interested in that, google Plackett-Burman.