Expectation of Random Variables
Certainly! The expectation (or expected value) of a random variable is a fundamental concept in probability and statistics, providing a measure of the "average" or "center" of a random variable's distribution. Here's how the expectation is defined for the three types of random variables:
Discrete Random Variables: A discrete random variable is one that can take on a finite or countably infinite number of distinct values. Let be a discrete random variable with possible values and let be the probability that takes on the value . The expected value of is defined as:
In other words, for each possible value of , you multiply that value by its probability and sum up all such products.
Continuous Random Variables: A continuous random variable is one that can take on a continuum of values over a certain interval or set of intervals. Let be a continuous random variable with probability density function (pdf) . The expected value of is defined as:
Essentially, you integrate over all possible values of , multiplying each value by its density, to get the average value.
General Random Variables: The definition of expectation for a general random variable (which could be mixed, having both discrete and continuous components) is a combination of the two above. In the most general sense, if is a random variable with cumulative distribution function (cdf) , its expectation is given by:
This definition encompasses both discrete and continuous cases as special instances.
It's worth noting that for a random variable to have a finite expectation, the sums or integrals above must converge to a finite number. If they do not, the expectation is said to be infinite or undefined.
A continuous random variable is one that can take on an uncountably infinite number of values in an interval (or multiple intervals). This is in contrast to a discrete random variable, which takes on a finite or countably infinite number of distinct values.
Here's a precise definition:
Continuous Random Variable: A random variable is said to be continuous if there exists a non-negative function , called the probability density function (pdf), such that for any two numbers and with :
- is the probability that the random variable lies between and , inclusive.
- The function satisfies two main properties:
- for all .
- (i.e., the total area under the curve of over the entire real line is 1).
Unlike discrete random variables, the probability that a continuous random variable takes on any specific value is always zero. This is because the number of potential values in any interval, however small, is uncountably infinite, so individual points have no width on which to accumulate probability.
A classic example of a continuous random variable is one that follows the normal (or Gaussian) distribution, which has a bell-shaped probability density function. Another example is a random variable following a uniform distribution on the interval [0, 1], where any value between 0 and 1 is equally likely.
Certainly! Measure theory offers a rigorous foundation for integration and is essential when we talk about more general random variables, especially in the context of probability theory.
Let's delve into the measure-theoretic definition:
Probability Space: Let be a probability space, where:
- is the sample space.
- is a -algebra of subsets of (these subsets are called "events").
- is a probability measure defined on , assigning to each event a number between 0 and 1.
Measurable Function: Let be a (real-valued) random variable. We say that is measurable with respect to (or -measurable) if for every Borel set in , . In simpler terms, this means that the pre-image of any Borel set under is an event in .
Expectation: The expectation of a random variable is defined as: This is the Lebesgue integral of with respect to the probability measure . Note that the random variable serves as the integrand, and the integration is over the sample space with respect to the measure .
The expectation is well-defined whenever the integral is absolutely convergent, meaning:
When we discuss real-valued random variables on , this measure-theoretic framework helps ensure all the definitions and operations are rigorously grounded. The definition of expectation given here captures both discrete and continuous random variables as special cases and also extends naturally to more general random variables.
In measure theory, the integral of a function with respect to a measure (like ) is defined using the concept of simple functions and then extended to more complex functions. The integral of a random variable with respect to the probability measure over a sample space , written as , is defined using the Lebesgue integral.
Here’s a rough sketch of how the Lebesgue integral is defined:
Simple Functions: Start with "simple functions." A simple function is one that takes on a finite number of distinct values. For a simple function that takes on values over sets , respectively, the integral is:
Positive Measurable Functions: For a positive measurable function (meaning for all ), the integral of is the supremum (least upper bound) of the integrals of all simple functions that are bounded by :
General Measurable Functions: For a general measurable function that can take both positive and negative values, decompose into its positive and negative parts: Then can be written as . Now, the integral of is defined as: provided at least one of the two integrals on the right-hand side is finite.
The beauty of the Lebesgue integral is that it can handle functions that are more "irregular" than those that can be integrated using the classical Riemann integral. In the context of probability, the Lebesgue integral allows us to integrate a broader class of random variables with respect to a probability measure.
Certainly. When you referred to the integral with respect to , you were pointing to the cumulative distribution function (CDF) of a random variable . The CDF is defined as:
Given the cumulative distribution function of a real-valued random variable , the expectation can be formulated in terms of using the Lebesgue-Stieltjes integral.
For a general random variable with CDF , the expectation can be defined as: provided the integral exists.
Breaking it down further:
Positive Values: Consider the part of the integral for positive values: This captures contributions from all positive outcomes.
Negative Values: Now consider the part of the integral for negative values: This covers contributions from all negative outcomes.
Combining these parts, the expectation is:
The integral is the Lebesgue-Stieltjes integral, which is a generalization of the Riemann-Stieltjes integral in a way that is analogous to how the Lebesgue integral generalizes the Riemann integral. This definition of expectation allows for a unified treatment of both discrete and continuous random variables, as well as more general mixed cases.
The Riemann-Stieltjes integral provides a generalization of the standard Riemann integral. While the Riemann integral integrates a function with respect to the variable , the Riemann-Stieltjes integral integrates a function with respect to another function, say .
Here's a precise definition:
Let and be real-valued functions defined on a closed interval . The function is said to be Riemann-Stieltjes integrable with respect to on if there exists a limit: for some choice of in each subinterval , where is a partition of into subintervals: and denotes the mesh or maximum width of the partitions:
If such a limit exists, then is said to be Riemann-Stieltjes integrable over with respect to , and the value of the integral is given by the limit above.
Several points worth noting:
- When , the Riemann-Stieltjes integral reduces to the standard Riemann integral.
- If is of bounded variation and is continuous, then is Riemann-Stieltjes integrable with respect to over .
- The integral can be thought of as "summing" the values of in a weighted manner, where the weights are determined by the increments of the function .
The Riemann-Stieltjes integral provides a useful bridge between discrete sums and continuous integrals, and it plays a foundational role in the development of more general integration theories like the Lebesgue-Stieltjes integral.
The Riemann-Stieltjes integral can be used to define the expected value of many random variables, but it's not sufficient for all cases, especially in the context of more complex or irregular probability distributions.
For discrete random variables and absolutely continuous random variables (those with a probability density function), the Riemann-Stieltjes integral is adequate for defining the expected value. Specifically:
Discrete Random Variable: If is a discrete random variable with possible values and probabilities respectively, then its cumulative distribution function (CDF) has "jumps" at these values. The expected value can be written as a sum: Which can be viewed as a Riemann-Stieltjes integral with respect to .
Absolutely Continuous Random Variable: If has a probability density function , then its CDF is differentiable almost everywhere, and . The expected value can be expressed as a Riemann integral: But it can also be viewed as a Riemann-Stieltjes integral with respect to .
However, for more complex random variables, the Riemann-Stieltjes integral might not suffice. Especially problematic are random variables associated with singular distributions (neither absolutely continuous nor purely discrete, with CDFs that are continuous but not absolutely continuous). In such cases, the Lebesgue-Stieltjes integral, based on the more general Lebesgue integration theory, is required to properly define the expected value.
In conclusion, while the Riemann-Stieltjes integral is sufficient for a large class of random variables, it's not general enough to handle all cases. The Lebesgue-Stieltjes integral provides a more comprehensive framework for defining the expected value of a general random variable.