| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
Backprop.Learn.Regularize
Synopsis
- type Regularizer p = forall s. Reifies s W => BVar s p -> BVar s Double
- class Backprop p => Regularize p where
- l1Reg :: Regularize p => Double -> Regularizer p
- l2Reg :: Regularize p => Double -> Regularizer p
- noReg :: Regularizer p
- l2RegMetric :: (Metric Double p, Backprop p) => Double -> Regularizer p
- l1RegMetric :: (Num p, Metric Double p, Backprop p) => Double -> Regularizer p
- newtype RegularizeMetric a = RegularizeMetric a
- newtype NoRegularize a = NoRegularize a
- lassoLinear :: (Linear Double p, Num p) => Double -> p -> p
- ridgeLinear :: Linear Double p => Double -> p -> p
- grnorm_1 :: (ADT p, Constraints p Regularize) => p -> Double
- grnorm_2 :: (ADT p, Constraints p Regularize) => p -> Double
- glasso :: (ADT p, Constraints p Regularize) => Double -> p -> p
- gridge :: (ADT p, Constraints p Regularize) => Double -> p -> p
- addReg :: Regularizer p -> Regularizer p -> Regularizer p
- scaleReg :: Double -> Regularizer p -> Regularizer p
Documentation
type Regularizer p = forall s. Reifies s W => BVar s p -> BVar s Double Source #
A regularizer on parameters
class Backprop p => Regularize p where Source #
A class for data types that support regularization during training.
This class is somewhat similar to Metric Double
Often, this doesn't include bias terms (terms that "add" to inputs), and only includes terms that "scale" inputs, like components in a weight matrix of a feed-forward neural network layer.
However, if all of your components are to be regularized, you can use
norm_1, norm_2, lassoLinear, and ridgeLinear as sensible
implementations, or use DerivingVia with RegularizeMetric:
data MyType = ... deriving Regularize via (RegularizeMetric MyType)
You can also derive an instance where no are regularized, using
NoRegularize:
data MyType = ... deriving Regularize via (NoRegularize MyType)
The default implementations are based on Generics, and work for types
that are records of items that are all instances of Regularize.
Minimal complete definition
Nothing
Methods
rnorm_1 :: p -> Double Source #
Like norm_1: sums all of the weights in p, but only the
ones you want to regularize:
\[ \sum_w \lvert w \rvert \]
Note that typically bias terms (terms that add to inputs) are not regularized. Only "weight" terms that scale inputs are typically regularized.
If p is an instance of Metric, then you can set rnorm_1
= norm_1p, even
potential bias terms.
rnorm_1 :: (ADT p, Constraints p Regularize) => p -> Double Source #
Like norm_1: sums all of the weights in p, but only the
ones you want to regularize:
\[ \sum_w \lvert w \rvert \]
Note that typically bias terms (terms that add to inputs) are not regularized. Only "weight" terms that scale inputs are typically regularized.
If p is an instance of Metric, then you can set rnorm_1
= norm_1p, even
potential bias terms.
rnorm_2 :: p -> Double Source #
Like norm_2: sums all of the squares of the weights
n p, but only the ones you want to regularize:
\[ \sum_w w^2 \]
Note that typically bias terms (terms that add to inputs) are not regularized. Only "weight" terms that scale inputs are typically regularized.
If p is an instance of Metric, then you can set rnorm_2
= norm_2p, even
potential bias terms.
rnorm_2 :: (ADT p, Constraints p Regularize) => p -> Double Source #
Like norm_2: sums all of the squares of the weights
n p, but only the ones you want to regularize:
\[ \sum_w w^2 \]
Note that typically bias terms (terms that add to inputs) are not regularized. Only "weight" terms that scale inputs are typically regularized.
If p is an instance of Metric, then you can set rnorm_2
= norm_2p, even
potential bias terms.
lasso :: Double -> p -> p Source #
lasso r prnorm_1) in p to be either r if that component was
positive, or -r if that component was negative. Behavior is not
defined if the component is exactly zero, but either r or -r are
sensible possibilities.
It must set all non-regularized components (like bias terms, or
whatever items that rnorm_1 ignores) to zero.
If p is an instance of Linear DoubleNum, then you can set
lasso = lassoLinearrnorm_1
counts all terms in p, including potential bias terms.
lasso :: (ADT p, Constraints p Regularize) => Double -> p -> p Source #
lasso r prnorm_1) in p to be either r if that component was
positive, or -r if that component was negative. Behavior is not
defined if the component is exactly zero, but either r or -r are
sensible possibilities.
It must set all non-regularized components (like bias terms, or
whatever items that rnorm_1 ignores) to zero.
If p is an instance of Linear DoubleNum, then you can set
lasso = lassoLinearrnorm_1
counts all terms in p, including potential bias terms.
ridge :: Double -> p -> p Source #
ridge r prnorm_2) in p by r.
It must set all non-regularized components (like bias terms, or
whatever items that rnorm_2 ignores) to zero.
If p is an instance of Linear DoubleNum, then you can set
ridge = ridgeLinearrnorm_2
counts all terms in p, including potential bias terms.
ridge :: (ADT p, Constraints p Regularize) => Double -> p -> p Source #
ridge r prnorm_2) in p by r.
It must set all non-regularized components (like bias terms, or
whatever items that rnorm_2 ignores) to zero.
If p is an instance of Linear DoubleNum, then you can set
ridge = ridgeLinearrnorm_2
counts all terms in p, including potential bias terms.
Instances
l1Reg :: Regularize p => Double -> Regularizer p Source #
Backpropagatable L1 regularization; also known as lasso regularization.
\[ \sum_w \lvert w \rvert \]
Note that typically bias terms (terms that add to inputs) are not regularized. Only "weight" terms that scale inputs are typically regularized.
l2Reg :: Regularize p => Double -> Regularizer p Source #
Backpropagatable L2 regularization; also known as ridge regularization.
\[ \sum_w w^2 \]
Note that typically bias terms (terms that add to inputs) are not regularized. Only "weight" terms that scale inputs are typically regularized.
noReg :: Regularizer p Source #
No regularization
Arguments
| :: (Metric Double p, Backprop p) | |
| => Double | scaling factor (often 0.5) |
| -> Regularizer p |
L2 regularization for instances of Metric. This will count
all terms, including any potential bias terms.
You can always use this as a regularizer instead of l2Reg, if you want
to ignore the default behavior for a type, or if your type has no
instance.
This is what l2Reg would be for a type p if you declare an instance
of Regularize with rnorm_2 = norm_2ridge
= ridgeLinear
Arguments
| :: (Num p, Metric Double p, Backprop p) | |
| => Double | scaling factor (often 0.5) |
| -> Regularizer p |
L1 regularization for instances of Metric. This will count
all terms, including any potential bias terms.
You can always use this as a regularizer instead of l2Reg, if you want
to ignore the default behavior for a type, or if your type has no
instance.
This is what l1Reg would be for a type p if you declare an instance
of Regularize with rnorm_1 = norm_1lasso
= lassoLinear
Build instances
DerivingVia
newtype RegularizeMetric a Source #
Newtype wrapper (meant to be used with DerivingVia) to derive an
instance of Regularize that uses its Metric instance, and
regularizes every component of a data type, including any potential bias
terms.
Constructors
| RegularizeMetric a |
Instances
newtype NoRegularize a Source #
Newtype wrapper (meant to be used with DerivingVia) to derive an
instance of Regularize that does not regularize any part of the type.
Constructors
| NoRegularize a |
Instances
Linear
Generics
grnorm_1 :: (ADT p, Constraints p Regularize) => p -> Double Source #
grnorm_2 :: (ADT p, Constraints p Regularize) => p -> Double Source #
glasso :: (ADT p, Constraints p Regularize) => Double -> p -> p Source #
gridge :: (ADT p, Constraints p Regularize) => Double -> p -> p Source #
Manipulate regularizers
addReg :: Regularizer p -> Regularizer p -> Regularizer p Source #
Add together two regularizers
scaleReg :: Double -> Regularizer p -> Regularizer p Source #
Scale a regularizer's influence