Email this to a colleague

Most continuing education departments have a firm grasp of our industry’s fundamental metrics: revenue, net revenue, enrollments, evaluation scores, and a few others. However, measurement beyond this is generally beyond the capability of continuing education staff. Few departments translate good for-profit business practices into a working continuing education model, and even fewer find such translations effective.

While it is difficult to derive some common business key performance indicators in the continuing education industry, they are not out of reach. It is critical, particularly in light of recent economic conditions, for lifelong learning and continuing education organizations to begin employing the same business practices as successful for-profit businesses.

This paper documents our efforts to create a new set of business metrics. Borrowing from both marketing and web analytics, we have designed key performance indicators for the continuing education industry. In this paper, we will introduce these metrics, demonstrate their value, and lay the foundation for a data-driven decision support system that identifies problem areas and suggests opportunities within course offerings. The metrics presented in this paper are innovative, and the resulting decision support tool is both unique and groundbreaking for continuing education.

Retention, retention, retention

Convincing existing students to take additional classes is a core part of most continuing education programs’ marketing strategy. It is cheaper and easier to generate additional business from existing customers than to find new customers. In addition, improving customer retention also improves customer lifetime value, making it easier to hit revenue goals with fewer students.

The two metrics, repeat rate and your retention rate, are similar, and they are often confused for one another. Repeat rate measures how often a customer repeats or takes more classes. Retention rate, on the other hand, measures how long a customer keeps doing business with your organization. Generally, a high repeat rate implies a high retention rate, but this is not necessarily the case.

Why repeat rate is critical to improve

While this paper focuses on repeat rate, customer lifetime value (CLV) should always be in the back of a continuing educator’s mind. If you increase the CLV of students, you can justify larger marketing budgets and still increase profits. To understand how to improve CLV, we need to understand how to compute it. Here is a formula commonly used to estimate CLV for our industry: Customer Lifetime Value equals the average income per registration divided by one minus the average repeat rate.

From this formula, we can see that there are two ways to increase CLV.

1. Increasing the income per registration without changing the average repeat rate
2. Increasing the repeat rate without decreasing the average income per registration

The first option is viable. This is how most businesses control the value of a customer. Increased prices produce more revenue from the same customers. However, increasing prices may also reduce your number of customers and/or reduce your average repeat rate.

On the other hand, there are many opportunities to increase repeat rate without increasing costs. Furthermore, efforts to increase repeat rate rarely carry significant risks. Therefore, improving repeat rate is usually a less risky approach to improving CLV than increasing prices.

How to improve Repeat Rate

The list below is describes several characteristics that affect your retention.

More dynamic course offerings increase your repeat rate. The more often your course offerings change, the more likely former participants will return for other classes.

More diverse course offerings increase your repeat rate. Offering a broad range of subjects in your course offerings increases the chance that participants will find something else that interests them.

Creating clear, logical groupings of courses will increase your repeat rate. For example, certificate programs with many electives, or series of courses (level 1, level 2, etc.).

Overall customer satisfaction also has a huge influence on retention, since happy customers are more likely to return for more classes.

The challenges of repeat rate

Repeat rate is a valuable concept for our industry. However, repeat rate, particularly in continuing education, has limitations.

Limitation #1: One size does not fit all.

Retention and repeat rate are most applicable in business models that deliver the same or similar products or services multiple times to the same customer. In continuing education, if a customer takes the same class multiple times, we are probably not doing our job well. Thus, traditional approaches to retention rate fail in the continuing education industry. In fact, for us, there is an unknowable, maximum repeat rate. Therefore, there is also a maximum retention rate.

Eventually, every student will graduate or will have taken all the classes that interest them. At that point, the student will cease being a customer. Thus, in a sense, if we are doing our job correctly, our customers should defect eventually. When they get their training, move on to a new career or new promotion, we cannot do more for them. Those blows to our retention rate are acceptable.

Limitation #2: It is difficult to derive or compute in continuing education.

Rate of return, retention rate, or repeat rate are all very difficult to compute in our industry. Customers often wait months or years before enrolling in additional classes. This makes it almost impossible to calculate the current retention rate for students.

Unlike other industries, such as the cell phone industry, where customers pay consistent fees at regular intervals, we have difficulty estimating the interval to use when considering repeat rate and no viable way to calculate recent retention—we are forced to always measure retention of the past. In other words, if a cell phone customer stops paying their monthly bill, we can assume they are no longer a customer: They defected to a competitor, or they have given up their cell phone. However, if a student does not register for a new class this month, they may just be delaying their next registration. Even year-over-year, calculations of repeat rate leave a lot of room for interpretation and error.  

Limitation #3: It is difficult to get useful numbers today about today.

Repeat rate is by definition a lagging indicator. We must ignore the students who are taking their first classes right now, because it may be weeks or months before we know whether that student will register for additional classes. If we do account for these students—who have not yet had ample time to repeat—our repeat rate will measure low.

The following chart shows how repeat rate increases over time for a given sample of new customers. Notice that it reaches some unknown maximum. This chart does not include new customers that come in as the studied customers continue to take extra classes. The lower line represents the repeat rate that you compute. Notice that the blue line is always higher; at any particular period (red line), the repeat rate that you measure is actually representative of the repeat rate in a previous period (blue line).

Limitation #4: There is no measure of order or direction.

Retention and repeat rate are unaffected by the possible sequences in which customers took classes. Without this information, we are unable to identify loss leader programs and courses in offerings. In other words, without measuring order, we cannot identify which classes start registration “streaks” and which classes kill registration “streaks”.

Limitation #5: One unusual customer can skew the data for everyone.

“Super customers”, who take many classes, may skew retention rates. Consider a customer who, after completing a management class, goes on to take 15 other classes. Traditional methods for computing retention will include all of those classes for which he registered. This may lead you to believe that your management class has a good retention rate, even though none of the other customers repeated. In this case, the “super customer” is inflating your retention rate.

A need for more useful metrics

The last limitation, the concept of the “super customer”, is enough on its own to warrant creation of a different metric. Customers who take many classes are generally different demographically from those who only take one. These super customers are worth pursuing as a target market. Unfortunately, the way we compute retention rate and repeat rate, we cannot identify different groups of customers. We need a new metric that can identify super customers and other types of customers.

Web Analytics

From the moment the internet became a valuable tool for business, organizations began tracking information about websites: hits, visits, conversions, etc. Since then, web analytics has developed tremendously, and analytics tools, such as Google Analytics, update with new features regularly.

The term web analytics refers to a set of metrics that measure visitor behavior on your website. These metrics include things like conversions, visitors, and visits.  

Web analytics can be very granular. We can use them to study specific behaviors of website visitors: once someone visits your site, which web page is she most likely to look at first? How long will she stay there? How many pages will she browse through before finally leaving? We hope to apply these qualities of web analytics to continuing education.

Web analytics inspired the development of the key performance indicators (KPIs) presented in this paper. As we present each KPI, we will discuss the parallel concept in web analytics for comparison.

Continuing Education Retention Analytics

The development of the Continuing Education Retention Analytics (CERAs) has one objective: to address the shortcomings of retention rate as a metric. We mean for CERAs to complement, not replace, measures of retention. After all, retention rate relates closely to customer lifetime value, one of the most important metrics in the business world. In addition, because CERAs tie-in with repeat rate, they relate to lifetime value. However, we will show that they are easier to understand, to compute, and to act on.

In developing the CERAs, we established several requirements that each would meet.

Requirement #1: They must be conceptually simple and intuitive.

It is easy to get your mind around retention rate. If someone buys from you again, s/he repeated; if not, s/he defected. CERAs must also be intuitive and easy to explain.

Requirement #2: They must be measurable and easy to compute.

CERAs must be something that any person having access to the appropriate data and a basic grasp of statistics and spreadsheets can reproduce. The information used to compute each CERA must be available through major registration systems and other common sources. We must be able to easily identify and measure specific actions taken by customers that contribute to the CERA.

Requirement #3: They must be relevant and relate to lifetime value.

For CERAs to be useful, they must have a mathematical relationship to repeat rate and customer lifetime value. We need to be able to use CERAs to increase our repeat rates and improve customer lifetime value. If it is possible to effect a change in a CERA without seeing a similar change in retention or lifetime value, then the CERA is not a useful metric.

Requirement #4: They must be sensitive to order.

Learning and development is a process, and understanding student behavior requires a concept of order. CERAs must help us discover where customers start and end their learning and development. They should also give us an idea of what they do in between courses.

Requirement #5: We must be able to act upon them immediately.

A low repeat rate could have many causes; thus, it is difficult to know how to respond to a low repeat rate. Maybe there are not enough offerings for a segment of customers. Perhaps you are not promoting related classes well. Maybe it is something we have not yet thought of. Repeat rate alone does not reveal the cause of problems. CERAs must do better: they must provide an idea of what we can improve. Perhaps most importantly, we must be able to see the current statistics, so we can make important and meaningful decisions about today’s numbers today.

Requirement #6: They need to be sensitive to different segments of customers.

It is important for the CERAs to help us identify different groups of customers when retention cannot. Retention does not allow us to differentiate between the following two classes:

1. A case in which nine customers take one class and one takes eleven classes
2. A case in which ten customers take exactly two classes

In particular, the second class is full of customers who are eager to get more from your programs, and you should provide them with more opportunities.  The first class is full of customers who could take more; most of them just did not want to.

Putting all these requirements together

Putting all of these requirements together creates a better version of repeat rate. We can easily capture the same amount of information, in a concise, intuitive, and meaningful set of metrics. When retention is low, CERAs will help pinpoint reasons why. When we see bad indicators, we can react to them immediately to produce meaningful improvements. They will also be easy to communicate to other team members and to other teams. Moreover, using them will save time and effort.

Introducing the CERAs

Now that we have adequately discussed the necessary characteristics for the CERAs to possess, we must develop each of them. Below, we present each of the three CERAs, discussing the motivation for its development, potential use for the CERAs, and an example to illustrate the concept for that particular CERA.

Bounce Rate

In web analytics, a “bounce” occurs when someone visits a web page and does not visit any other pages on the same site. In other words, a person arrived and then immediately left. Many web improvement efforts focus on reducing bounce rate; the more pages a person views on a website, the more likely they are to convert to a customer.

In continuing education, we can define a bounce similarly. When someone takes only one course, we say that s/he has “bounced” from your program. Thus, a particular course’s bounce rate (BR) is the proportion of students who took that either course, but no other courses— before or after— compared to those who took that course and other courses:

Since bounce rate (BR) measures how well a class retains business, it relates closely to retention rate; however, it is both easier to compute and act upon than a traditional retention rate as we will demonstrate below

A BR of 0% means that everyone who took a particular course also took at least one other course. A course required as part of a certificate program would have a low BR since enrollees are required to take other courses as well. A BR of 100% means that no one who took a course took any other courses. This is a common occurrence for offerings of certain types. For example, a single course that offers training for a specific career should have a bounce rate close to 100%. The customers came for job training and ideally moved on to a new career.

BR measures two things about a course simultaneously:

1. How well the course encourages registrants to take additional courses
2. How well other courses encouraged registrants to sign up for the course

Both are critical for improving your repeat rate; therefore, your focus should be to decrease BR.

BR is simple to compute, yet a powerful indicator of opportunities to improve repeat rate. Sometimes the simplest KPIs can offer more insight than those requiring complex calculations. BR is a simple way to divide your customers into two fundamentally different groups:  those who only take one course and those who take two or more. Compared to customers who only take one class, we need to communicate and market differently to them than those already inclined to take multiple offerings, and BR facilitates this.

Furthermore, there are natural groupings to courses and certificates that encourage registrants to take a defined set of courses. This creates a push approach to marketing—you tell your potential customers what they want. On the other hand, by studying registration patterns of your customers, you can apply lessons from pull marketing—you let customers tell you what courses they want to take together—to enhance your cross advertising. In a future whitepaper, we will share a process to analyze registration patterns and interpret them.

How to use BR

BR can identify easy opportunities for improving repeat rate. For example, if the BR is already high for courses with natural follow-ups, e.g. Excel Introduction and Excel Intermediate, then advertising the intermediate course to participants in the introduction class may improve repeat rate. Renaming related courses may help as well. For example, consider a Delegation course and a Supervision course that target similar audiences, but have high bounce rates. By modifying their titles to create a sense of connectedness between the two (e.g. Delegation for Managers and Supervision for Managers) could lower the bounce rates for both.

Many enrollees in courses with low BR have a degree of loyalty to your program (since they are repeat customers). Thus, BR offers a quick way to identify opportunities to market to your best customers. These people are more likely to respond to follow-up advertisements and recommend your organization to their friends, family, and co-workers. BR can help identify these “champions” of your programs.

Title	BR
Essentials of Performance Management	24%
Essentials of Motivation	25%
Essentials of Counseling: Managing Difficult Employees	35%
Executive Presence for the Non-Executive	58%

Finally, you can use BR to determine likely success of new offerings. Building a new offering to complement a course that has both high enrollments and low BR increases its chance of success. BR can take some guesswork out of which new courses to develop. Use BR to identify opportunities to grow and diversify your offerings.

An example

A low BR means a higher repeat rate and CLV for customers taking those courses. Below is a table of management courses at Emory University and their respective BRs. All of the courses are electives in the same certificate program. The first three courses share a similar naming convention; they all start with “Essentials of”. While we can’t say for sure, their consistent titles are probably improving the BR by encouraging students that take one of the three classes to enroll in the other two.

On the other hand, Executive Presence for the Non-Executive (a popular course) has a markedly higher bounce rate than the other three. This class covers topics such as communication skills, presentation skills, and leadership skills. Enrollees in the Executive Presence course are clearly interested in other topics offered, but less than half take more courses.

This illustrates two points:

1. Course titles may affect customer perception of what classes you intended for them to take together.
2. Popular courses with high bounce rate are lost opportunities to increase retention rate.

BR versus repeat rate

Before moving on to the other CERAs, it is important to illustrate some of the value that BR provides on its own. In comparison to repeat rate, BR demonstrates itself to be at least as superior a metric in three major ways. We describe each below.

BR provides a natural way to segment customers

The simplicity of BR offers advantages in customer segmentation. While repeat rate provides a measure of probability that a customer will repeat, BR divides the population into two definitive groups: customers who took only one class and customers who took more than one class. Consider the two following scenarios in which we analyze three students:

• Students A and B took only one class. Student C took three classes.
• Students A and B took only one class. Student C took five classes.

In both cases, the BR is the same, but in the second scenario, the repeat rate is higher. Thus, the BR is unaffected by the “super customer” unlike the repeat rate. Generally, customers who took more than one course (non-bouncers) either:

• Are attempting to complete a certificate program by taking electives
• Already got the training they needed, and thus were already well informed of your listing
• Are already members of your “captive” audience

Non-bouncers are “elite” customers, deserving a different marketing and communication approach. BR more easily distinguishes this group from one-time customers.

Repeat rate differentiates between different types of non-bouncers, i.e. those who took two courses versus three, three courses versus four, etc. However, repeat rate does not juxtapose the different frequency of customers: those who took one course, those who took two courses, etc.; hence, repeat rate does not tell you how many customers are of what type. BR tells you exactly how many were bouncers and how many were non-bouncers. Thus, BR divides your customers into two groups with different needs.

BR is easier to compute

Despite the importance of repeat rate in continuing education and lifelong learning, it is a slippery metric. The typical approach to computing repeat rate does not easily apply in our industry and, in fact, can lead to incorrect assumptions.  It emerges from businesses that offer products or services customers buy repeatedly. In continuing education, however, students rarely repeat the same course and many only want to take one course.

BR is a better metric. BR is easier to compute than repeat rate. Computing a reliable repeat rate involves determining a cut-off date after a registration, analyzing a time-ordered list, and counting individual registrations. For a list of more than 10,000, this can take a computer a long time to compute, not including all of the querying, analysis, and interpretation on your part. BR, on the other hand, is a simple division of the number of registrants who only took one course by the total number of registrations.

Furthermore, we can compute BR at different levels of granularity. You can compute BR for all of your programs, one program, a collection of courses, or a single course. Think about repeat rate. Should you count repeats that cross from professional to personal enrichment courses? Should you count repeats from certificate to non-certificate courses or vice-versa?

BR is inversely related to repeat rate

Decreasing the number of customers who “bounce” simultaneously increases the number of returning customers. Hence, decreasing the BR automatically increases your repeat rate. The figure below demonstrates this relationship.

Furthermore, because of the relationship between BR and repeat rate, you can use BR to compute a lower bound —best conservative estimate—for repeat rate. Since BR is the percentage of customers who bounce, (100% – BR) is the percentage of customers who repeated. By definition, all repeat customers have taken at least two classes. Therefore, your actual repeat rate is at least as large as the (100% – BR) figure.

You can use repeat rate to compute a true ROI for marketing engagements. Using this best conservative estimate can give you a safe approximate ROI for decision-making. If you use BR in this way, you can expect repeat enrollments to generate at least as much revenue as the BR predicts. Thus, you can use BR in place of repeat rate as a KPI.

CERA #2: Landing Rate

Landing rate (LR) is a measure of how effectively a class attracts new business. In effect, it identifies the first contact between customers and your offerings.

We drew this concept from web analytics. In web analytics, we can measure the effectiveness of advertisements with landing pages. Online marketing often drives potential customers to specific landing pages. You can then measure how successful the advertisement and landing page are by comparing the number of visitors who complete a purchase, the number of people who never went to the landing page, and the number who did not purchase after arriving at the landing page.

In continuing education, some courses are more likely to attract new customers than others. For example, Microsoft Excel: Introduction will typically draw more new students than Microsoft Excel: Intermediate, since many Intermediate students would have previously taken the introductory course.

Every customer in your registration system has a “landing course” which represents the first class taken. Using this information, we can calculate the likelihood of a specific course in generating new enrollments as follows.

An LR of 100% means that every enrollee in a particular course is a new customer, while a LR of 0% means that everyone who takes a course was a returning student. Certificate capstone courses usually have LRs near zero, since participants must have completed several prerequisite courses.

LR measures one important thing about a course, program, or instructor: How well the course brings in new business. New registrants need the most encouragement from your marketing to take even more classes, so courses with a high LR are opportunities for targeted marketing.

Just like BR, LR is another way to measure the behavior of your customers.

How to use LR

LR is not always useful, since we reasonably expect advanced courses and capstones to have a LR near 0%. Statistically, it is impossible for most of your classes to have a high LR, unless your repeat rate is near zero. Therefore, focus on improving the LR for particular courses that should be bringing in new customers, such as introductory courses or loss leaders. Such courses can be called “landing courses”. 

In combination with bounce rate (BR), you can determine a course’s effectiveness as a loss leader that generates new repeat business. Ideally, introductory classes should have a high LR and a low BR. In non-technical language, that means you would hope these classes would collect many new customers (high LR), and most of those new customers would return for more classes (low BR). Since LR tells you how well a course generates new business, a landing course is exposes many new students to your organization. If the BR for a landing course is low, you can afford to make less on such courses, since they attract new customers that go on to take other courses. Consider discounting such courses.

An example

Below is a table showing courses from Emory University and their respective BRs and LRs. These four cases represent common combinations of LR and BR. Career Assessment and Exploring Entrepreneurship are introductory classes or loss leaders for other courses in Emory’s program. The other two classes are IT classes, and represent two disparate classes when it comes to LR and BR.

Title	LR	BR
Career Assessment	94%	54%
Exploring Entrepreneurship	87%	34%
Cascading Style Sheets	19%	11%
MS Project 2003	90%	82%

Ideally, the Career Assessment class should act as a landing course. We would hope as program managers that a lot of new business would come through this class. Many of the students, logically, continue taking classes to improve resumes, interview technique, or develop new marketable skills. In this case, the LR is 94%, which is very high. However, the BR is 54%. Almost all the customers taking Career Assessment are new, but only half take additional classes. With a LR of 84%, most of the business generated by Exploring Entrepreneurship is also new business, but with a BR of 34%, almost two-thirds of the customers take more courses. Exploring Entrepreneurship, therefore, represents a better loss leader than Career Assessment.

With Cascading Style Sheets and MS Project 2003, something interesting is happening. For Cascading Style Sheets, a 19% LR suggests that enrollees in this class consist primarily of repeat customers. However, an 11% BR means that most Cascading Style Sheets took another class before or after. In general, courses that have low LR and BR are probably part of a certificate program or a typical set of classes customers take together. Cascading Style Sheets is contributing to a high repeat rate, in this case.

MS Project, on the other hand, caters to a niche. Its high LR (90%) means that nearly all of the students are new business. The corresponding high BR (82%) means that nearly none of those new students take additional classes. Programming staff should question why this is the case. Is there no more training that these students need? Can you identify more training for these students? MS Project, in this case, is dragging the repeat rate down for its entire program.

CERA #3: Exit Rate

The inverse of the LR, exit rate (ER) is a CERA that identifies classes that tend to be the last a customer takes. ER measures the last contact between customers and your offerings.

In web analytics, exit rate measures how many people left the site after viewing a particular page. In continuing education, we can use a similar concept. A customer “exits” when he or she stops doing business with your organization, whether the customer gets training elsewhere or simply does not see the need for more training. Every customer has an exit class; even those who bounce (took only one class). To calculate the ER for a course, we count the number of exits for that course and divide it by the total number of enrollments.

An ER of 0% means that everyone in that course enrolled in another course. An ER of 100% means that every enrollee in a course stopped doing business with you after completing that course. We can expect low ER for any course before a capstone in a certificate series. You might expect capstones to have ER of 100%, but this is not necessarily the case. Capstone courses for a certificate program should have higher ER than other courses in the same certificate, but even students earning a certificate may have other educational needs. We can help lower ER for offerings by having follow-up opportunities for all classes.

How to use ER

Unlike landing rate, which is not a useful metric for all types of courses, ER is always valuable. Like BR, we can use ER to identify opportunities to improve repeat rates. Your objective should always be to decrease high ER. Identify courses with a high ER; those courses are opportunities to improve repeat rate and increase customer value.

From our earlier discussion of course bounce rate (BR), we know that decreasing the BR improves repeat rate and customer value. However, courses with low BR may still be opportunities to improve customer value. A high BR always results in a high ER, because the bouncers are also exits. However, a low BR does not necessarily result in a low ER, since there are two possible causes for a low BR:

1. The LR is high and the ER is low
2. The LR is low and the ER is high

In the first case, the course is performing efficiently. This course generates a lot of new business (high LR), and that business continues to do more business (low BR and low ER). In the second case, while most of the enrollees are repeat students (low BR and low LR), many stop doing business after that course (high ER). We should strive to turn all of the type 2 classes into type 1 classes.

An example

Below is a table of IT courses at Emory University and their respective ERs and BRs. The Photoshop: Web Graphics and QuickBooks classes serve two different audiences, based on how the programs are constructed. The two MS Project classes provides an interesting comparison.

Title	ER	BR
Photoshop: Web Graphics	33%	10%
QuickBooks	89%	67%
Project 2003: Intermediate	87%	21%
Project 2003: Introduction	57%	50%

The Photoshop class, with low ER and low BR represents an ideal class type. This class is part of a certificate program, so the number of repeaters for this class is high. Therefore, students in this class have a high value to the organization. In contrast, QuickBooks has a high ER and high BR. It caters to a narrow group of customers that rarely enroll in other offerings. Likewise, students who take other classes do not often take QuickBooks. Therefore, QuickBooks customers have lower relative value than Photoshop customers do.

The Project 2003 classes are two different types of classes as well. Clearly, the Introduction class leads to many enrollments in the Intermediate class. The Intermediate class’s low BR reflects this trend—these customers are not bouncers. The ER for both classes, however, shows opportunities to improve the repeat rate and customer value. With an ER of 57%, more than half of the customers in the Introduction class never return. Presumably, many of the customers who do repeat are taking the Intermediate class, which has a very high ER of 87%. We can improve the value of the Introduction class customers by more effectively encouraging enrollment in the Intermediate class. Likewise, we can increase the value of the Intermediate class customers by providing additional follow up classes.

CERA in Summary

The last three sections discussed CERAs used to improve the value of customers and to increase repeat rates for any continuing education or lifelong learning organization. All of the metrics, bounce rate, landing rate, and exit rate, are flexible enough to apply to an individual class, a group of classes, or an instructor’s classes. They are also simple to compute—each a simple ratio.

Each of the three CERAs, individually, tells part of a story about your course offerings. In combination, they provide a full picture of the role of a particular course or set of courses in your program. Together, the three metrics allow you to pinpoint opportunities to improve the repeat rate and lifetime value of your customers in a way no one in our industry has ever done.

Using Course Retention Analytics and Income to Facilitate Decisions

This section presents a reliable way for open enrollment continuing education program staff to use uses financial data and continuing education retention analytics (CERA) to make tough decisions about classes or groups of classes. This data-driven model suggests when to change courses, develop similar courses, and even when to stop offering courses. This forms the basis for a decision support system (a tool that automates or augments a data-driven decision-making process) using empirical course data to make recommendations about courses.

Dimension one: CERA archetypes

The first step in constructing the model is to develop a convention for using CERAs to define course archetypes. These archetypes are the first dimension we need to support our program staff’s decision-making process.

Every course will have a bounce rate, landing rate, and exit rate. Together, these three metrics can give us a comprehensive idea of where a course fits into a customers’ entire learning experience. Is it likely to be the first class? Is it likely to be the last class? Is it somewhere in between?

Courses with a high landing rate, a high exit rate, and a high bounce rate are solitary courses. Enrollees in these courses rarely take any other courses. Courses with a high landing rate, but a low exit rate are entry courses, since they lead to registrations for other courses. Courses with high exit rates and low bounce and landing rates are exit classes, as they mark the end of a customer’s interaction with your programs. Connection classes are classes that fit somewhere between the entry and exit classes in the learning experience. The following table summarizes all possible types of courses based on these three metrics.

Bounce rate	Exit rate	Landing rate	Archetype name	Enrollees in this class…
Low	Low	Low	Connection class	took other classes & will take more
Low	Low	High	Entry class	started in this class; they will take more
Low	High	Low	Exit class	took other classes; will not take more
Low	High	High	Two-way class	entered (and will take more) or exited (and took others)
High	Low	Low	N/A	Does not exist
High	Low	High	N/A	Does not exist
High	High	Low	N/A	Does not exist
High	High	High	Solitary class	did not take other classes & will not take more

Note that some combinations simply are not possible. For example, a course can never have a high bounce rate and low exit and landing rates since a bounce is automatically an exit and a landing. To determine what archetype a class is, simply compute the CERAs for that class, and look up the values in the table. We define each of the archetypes below.

• Connection class—these classes tend to neither be the first, nor last that a student takes. They have low entrance rates and low bounce rates. People in these classes will more than likely go on to take more classes, and they took something before this class.
• Entry class—these classes tend to be at the beginning of a customer’s interaction with your program. Most of the people enrolled in these classes have not yet taken any other classes with you. They are also likely to take more classes afterward.
• Exit class—while people in these classes will typically never come back to take more classes, they did take classes before this one. We should expect there to be some exit classes, particularly in a certificate program or in a natural series of classes (e.g. level 1, level 2, level 3 type setups).
• Two-way class—these classes seem strange if your offerings are very structured. However, if students typically look at your offerings and can choose electives in any order, they are a natural occurrence. For some students in this class, this is their entry class; for other, this is their exit class. In both cases, this class will not be taken only once.
• Solitary class—these classes are standalone classes. People who take this class are likely to both make it their first and last class they take with you. This is something we could expect under certain circumstances. In general, however, solitary classes may represent a missed opportunity to develop new curriculum.

Dimension two: Financial data

The second dimension in this model involves measuring the profitability of a course. While many continuing education units struggle with track indirect expenses for specific course, this is necessary for this model. If this information is unattainable for your programs, you may approximate profitability.

There are four distinct categories of profitability for a course. They are presented below in order of increasing profitability:

1. Money-loser: This course’s expenses outweigh the revenues.
2. Break-even: This course’s expenses equal (or come close to equaling) the revenues.
3. Profitable: This course’s revenues outweigh the expenses.

Once we have identified the financial health of a course, we can combine this dimension with the CERAs.

Combining the Dimensions

Once you have the CERA archetype and profitability for a class, the model automatically suggests an action for that particular class. Keep in mind that these actions are recommendations. Always exercise good judgment and consider additional factors before making changes to your programs.

The following table lists all combinations of both dimensions. Simply look up the CRA archetype and the financial health of your class in the following table for the suggested action.

If CERA Archetype is…	And profitability is…	Then…
Connection class	Loss	Merge it with another class
Exit class	Loss	Merge it with another class
Entry class	Loss	Keep the class the same
Two-way class	Loss	Merge it with another class
Solitary class	Loss	Kill it—there is no market for this class
Connection class	Break-even	Update the class details
Exit class	Break-even	Update the class details
Entry class	Break-even	Keep the class the same
Two-way class	Break-even	Update the class details
Solitary class	Break-even	Update the class details
Connection class	Profitable	Keep the class the same
Exit class	Profitable	Keep the class the same
Entry class	Profitable	Keep the class the same
Two-way class	Profitable	Compliment the class with additional curriculum
Solitary class	Profitable	Compliment the class with additional curriculum

Combining these two dimensions establishes groundwork for the first decision support system specific to the continuing education industry. While the development of this tool requires more IT expertise than presented here, the concept is invaluable to your program staff. Even approximates of this data will help in day-to-day decisions continuing education staff make about running profitable course offerings.

Conclusion

Experience and experts tell us that repeat rate and customer lifetime value are important metrics. However, repeat rate and lifetime value, are both difficult to compute in continuing education. One student taking many classes can also inflate calculations of repeat rate.

Our model overcomes the limitations of repeat rate by constructing a new set of retention metrics specific to continuing education. As individual metrics, they provide actionable information about courses, instructors, and curricula. These metrics help us know how and where to make improvements.

By combining the three CERAs, we can archetype classes based on how each class coincides with the registration lifecycle of a customer. We can identify classes that tend to be at the beginning, middle, and end of students’ learning experience. In addition, we can identify classes that attract students who rarely take other classes, called solitary, standalone, or bounce classes.

Combining these archetypes with financial information breaks new ground in the continuing education industry, laying the foundation for an industry-specific decision support system. This system provides recommendations to maintain, improve, or eliminate classes using metrics derived from enrollment data.

Continuing education departments are increasingly incorporating data-driven business principles in their work. We hope this whitepaper helps improve the measurement, understanding, and application of repeat rate for the industry.

• Repeat rate is the proportion of registrations that result in an additional registration from the same student
• As suggested by other continuing education organizations such as LERN
• This could result in analytics problems later. If you market to customers who already intend on completing a certificate with electives, you may end up attributing what that customer was already going to take to your ROI analysis of marketing efforts, skewing your judgment.
• See JMH’s article "Four Reasons to Use Course Bounce Rate, Not Rate of Return”
• For a bouncer, the landing course and the exit course are the same course.
• There are two ideal classes. We can measure the performance of all courses using these three metrics. The ideal types of classes have either: (1) low BR, high LR, and low ER, or, (2) low BR, low LR, and low ER.
• Constitution of a high and a low value depends on the dataset. In general, 50% is a good cut-off, however, for landing rates. For bounce rates and exit rates, it is good practice to allow values to range slightly higher, like 60%. This is because after a class runs, there is always a chance that an enrollee is waiting to take another class. Thus, the exit rate and bounce rate are always high.