College Football Playoff: Lessons Learned from Evaluating Teams as a Member of the Mock Selection Committee

Grabven, Texas – I was a newspaper student at Baylor in 2014 when the final sketch on ESPN flipped to No. 4 in Ohio. The Bears, who competed in the Big 12 with TCU, found themselves sitting outside their first College Football playoff game. Eight years later, to my great dismay, the College Football Playoff Selection Committee did exactly the same thing.

But we will get to that.

Twelve members of the media, including myself, were invited to participate in the College Football Playoff’s annual photo selection. Rehearsals take place behind the doors of the same opulent boardroom at Gaylord Texan as the actual committee that holds the fate of countless sports departments in its hands.

On that day, she played the Navy’s athletic director and current CFP member, Chet Gladchuk. “Act a little cool to play Chet,” joked one of the 72-year-old CFP employees who took his seat. But if it’s all about melancholy, this controversial 2014 match rework was the perfect place to start.

The challenge of ranking the four ‘best’ teams

The team ranking process has been widely reported, but here’s a quick summary: The rankings are broken down into seven groups – three points 1-9 and all four 10-25. We vote to consider six teams simultaneously for each threesome. Group I: Alabama, Ohio State, Oregon, Florida, Baylor, University of California. As it should be.

CFP President and NC State Athletic Director Bo Corrigan tasked us with finding the top 25 teams in the country. But from the start, it became clear that we each have different definitions of this scale. Some valued success throughout the season, while others wanted the best team as of Selection Sunday. Regardless, resume ended up being a far more important consideration than which team would win on any Saturday.

Debating the standards of some of the most famous sports names was probably the part I wasn’t prepared for so much. Previous players in the room were adamant that Florida’s undefeated season was a trump card. I disagree. Try being a rookie like me when he told Pro Bowl running back Deuce McAllister and former first-round quarterback EJ Manuel that winning isn’t all due to metrics.

CFP has an analytics system created by SportSource Analytics that can compare up to four teams simultaneously with a wide range of metrics on giant screens in front of the group. When a Baylor vs. TCU vs. Ohio vs. Florida comparison appeared on screen to determine between points #3 and #4, everything that happened in 2014 made sense.

Jiarajah (left) and former soccer player Kirk Morrison

Kevin Jayraj/College Football Playoff

Team schedules are displayed on the page with a list of scores and opponents involved or head-to-head. However, the most striking part of the page is the color gamut with the difference marked from green (good) to red (bad). When the TCU and Baylor tables came out, the amount of red against the rest of the field was staggering.

Never underestimate the power of data visualization!

Furthermore, one of the only real criteria listed by the committee is to give precedence to similar teams with direct victories or conference tournaments. CFP Director Bill Hancock assured the group that since Baylor and TCU were both introduced as co-champions by the Big 12, each team should effectively be treated as holding 0.5 titles.

Compared to Ohio and Florida, the decision was not difficult. We can argue about the specific strength of the schedule and the rating scales the committee uses, but it gave us a complete picture of the committee’s decision.

Perhaps with guidance from Manuel State of Florida in the room, Noles actually jumped to number two in our fictitious rankings, behind only Alabama. I voted Florida No. 5; The twelve metrics identified by the Trade Competition Commission as most relevant to winning hated FSU, and his narrow winning margins weren’t enough to convince me of the other (best) conference champions.

After full disclosure, I tended to rely more on the efficiency and quality of play numbers than on other voters. SportSource’s meticulously structured, color-coded analytics which correlated the most with winning after extensive historical research, with relative scoring and play for every foul point leading the way. (A special teams metric was added to the panel’s 12 most important factors at the request of former Nebraska coach Tom Osborne.) Other members of the mock committee spoke of the quality of their schedule. Some have leaned on playing in the middle, or trying to bring in NFL odds.

Ultimately, this diversity of opinions is the goal. Seven of the thirteen CFP members are athletic directors. Seven former college players, including former NFL player and MIT mathematician John Urschel. Two coaches. One of them, former USA Today columnist Kelly Whiteside, is a journalist. Everyone sees the game differently.

The combination of past players, writers, and television personalities gave our group a similarly interesting context. San Diego State legend and NFL vet Kirk Morrison wanted to ensure that the group-of-five contenders got the attention they needed. Ari Wasserman of Athletic asked us to think about recruitment gaps. AL.com editor John Talty implored us to remember that Katy Perry was at The Grove for Ole Miss with Alabama. McAllister and former Michigan quarterback Devin Gardner got us thinking about winning above all else. You pointed out that the Ole Miss was lost 30-0 by the Britt Pilima Arkansas team that achieved the so-called “borderline sensational” with a win over Texas. Everyone has a role.

NC State Athletic Director and CFP President Bo Corrigan

Kevin Jayraj/College Football Playoff

How to arrange the best rest

If you are wondering why there are moments of inconsistency in the arrangement, this is why. Each secret ballot is characterized by different voters balancing their own criteria while trying to convince other voters that their view is correct. This really made the numbers 7-25 conversation infinitely more interesting and competitive.

“When I was ARMY’s athletic director, we were thrilled that we made it to the top 25,” Corrigan told the group. “We have to give the lowest ranking the same amount of attention as the top.”

This is what we did. The process required it.

Some attendees tried to project the power of the conference into the picture, or rejected some programs for being a G5. CFP Executive Director Bill Hancock and Corrigan quickly shot them. After delving into the matches and seeing the tab gradient from green to red again, it became frankly clear that this wasn’t necessary.

The data visualization had records compared to the difference above .500, against the previous top 25 of the committee and the previous top 10, but in truth, there were very few high-end data points to choose from. Instead, tables filled with green caught our attention. Ranked games – or border rated games – were talked about on a case-by-case basis.

We withdrew from Boise State and some were shocked by the amount of outstanding opponents at the “Group of Five” convention. Has stepped up. Conversely, the “Power Five” schedule in Wisconsin was filled with red. The SEC Ole Miss and Georgia contenders were filled with green wins and red losses. Michigan State and Kansas State had no green wins, but their only flaws came from the country’s top teams.

After each round, we would vote for new teams. In the end, we had the opportunity to discuss any significant discrepancies. UCLA was originally ranked behind the Arizona State team that had the Bruins hanging out in Tempe, Arizona. It has been corrected. Georgia moved further down, while Arizona went up several points. Marshall entered the field after comparing well with teams like Minnesota and Louisville in 12 Factors.

By the way, this discussion took place over a period of five hours. The Real Thing features six weeks of talking, endless hours of discussion and countless weekends of watching clips from every football related game imaginable. Hancock noted that there was one member of the committee who decided to rank each player individually in every game they watched. Every year is different.

What will change (and what won’t)

Naturally, this brings us to the 12-team playoff. In many ways, the decisions the commission makes are about to change. With six automatic bids, there will be less focus on knowing which teams are on the field and more focus on rankings and teams.

When the playoff round expands, the committee will remain at 13 members. They still plan to rank 25 teams and are comfortable that at least six conference champions will fall into that number. Instead of filling the pot pairs with the teams below the bracket, these groups will now fill the big playoff spots. Basically, the process won’t change much as the playoffs expand, which is what those in the room are hoping for sooner rather than later.

In the end, I gained a lot of respect for the process. It’s not perfect, it’s not without a doubt, but it is comprehensive. With the amount of information and video available to committee members, I feel more confident that they have all the tools to make the best possible decisions. More importantly, the secret ballot process makes it somewhat difficult to manipulate the order for branding or matching purposes. Transparency won’t quell any conspiracy theories, but it does make me more confident in the process.

And to Hancock, whenever I need a new name to add to my list of committee members…I’m available.