This document describes how you can use Cloud Monitoring to observe trends in your logs and to notify you when conditions you describe occur. To provide Cloud Monitoring with data from your logs, Logging offers you the following:
- Log-based metrics, which you can use as follows:
- To create alerting policies that notify you of changes over time.
- To create charts that display changes over time.
- Log-based alerting policies, which notify you anytime a specific event appears in a log.
When you want to monitor your logs over time, use charts or alerting policies based on log-based metrics.
When you want a notification, in near real time, that a message has appeared in your logs, use a log-based alerting policy.
The rest of this document describes how to choose between log-based metrics and log-based alerting policies, discusses the differences between them, and provides information about authorization, costs, and limits.
Log-based metrics
When you want to monitor recurring events in your logs over time, use log-based metrics. Log-based metrics generate numeric data from your logs. Log-based metrics are suitable when you want to do any of the following:
- Count the occurrences of a message, like a warning or error, in your logs and receive a notification when the number of occurrences crosses a threshold.
- Observe trends in your data, like latency values in your logs, and receive a notification if the values change in an unacceptable way.
- Create charts to display the numeric data extracted from your logs.
Because log-based metrics generate numeric data from your logs, you can use these metrics in alerting policies and display them in charts. For information about creating alerting policies and charts for log-based metrics, see Configure notifications for log-based metrics.
Cloud Monitoring provides a set of predefined log-based metrics, and you can define your own. To see a list of the system-defined log-based metrics, click the following add_circle button:
User-defined log-based metrics
You can create log-based metrics to extract numeric data from your logs. User-defined log-based metrics calculate values from both included and excluded logs.
By default, user-defined log-based metrics collect data from all logs received by the Log Router in your Google Cloud project, but you can define log-based metrics that collect data from logs routed to a specific log bucket.
- For information about defining and using project-level log-based metrics, see Using log-based metrics.
- For information about defining and using bucket-level log-based metrics, see Log-based metrics on log buckets.
If you define your own log-based metrics, you might incur charges. For more information about costs associated with metric ingestion, see Chargeable metrics.
Log-based alerting
When you want to be notified anytime a specific message occurs in a log, use log-based alerting policies. Log-based alerting policies are useful for catching security-related events in logs, like the following:
- You want to be notified when an event appears in an audit log; for example, a human user accesses the security key of a service account.
- Your application writes deployment messages to logs, and you want to be notified when a deployment change is logged.
Log-based alerting policies are useful for events that you expect to be both rare and important. You don't want to know about a trend or pattern; you want to know that something occurred.
For information about creating log-based alerting policies, see Using log-based alerting policies.
You can configure a metric-based alerting policy to simulate a log-based alerting policy. To do so, define a log-based metric and use the metric in a metric-based alerting policy with a threshold of one. Log-based alerting policies give you that behavior without the need to create a log-based metric and configure a metric-based alerting policy.
Comparison of alerting options
This section compares alerting policies built on log-based metrics to log-based alerting policies.
Summary table
The following table summarizes the alerting techniques and provides links to additional information in this document:
Incidents from log-based metrics | Log-based alerting policies | More information |
---|---|---|
Based on metrics derived from log entries | Based on strings in individual log entries | Log-based metrics and Log-based alerting |
Used to notify you of trends over time | Used to notify you when a specific message appears in a log | Log-based metrics and Log-based alerting |
Calculated from
|
Match only included logs | Available logs |
Operate on metrics from all projects in the metrics scope of the scoping project | Operate on logs in the scoping project only | Alerting across multiple projects |
Triggered when the value of a metric meets a condition for a specified period | Triggered each time a specific log entry matches a filter | Incidents and notifications |
Created and managed in Monitoring | Created in Logging; managed in Monitoring |
Creating and managing alerting policies |
Viewed in Monitoring | Viewed in Monitoring | Viewing alerting policies |
Can use any notification channel supported in Monitoring | Can use any notification channel supported in Monitoring | Notification channels |
Available logs
User-defined log-based metrics are calculated from all logs received by the Logging API for the Google Cloud project, regardless of any inclusion filters or exclusion filters that may apply to the Google Cloud project. If you create an alerting policy based on a user-defined log-based metric, then the policy monitors data from all logs.
System-defined log-based metrics are calculated only from logs that have been stored in log buckets in the Google Cloud project. If a log has been explicitly excluded, then it isn't included in these metrics. If you create an alerting policy based on a system-defined log-based metric, then the policy monitors data only from included logs.
Log-based alerting policies operate only on included logs originating in a project. You can't use log-based alerting policies to notify you about messages in excluded logs, or logs from other resources, such as folders or organizations.
Both log-based metrics and log-based alerting policies operate on the Google Cloud project scope, not on individual buckets.
Alerting across multiple projects
You can monitor metrics from several projects by configuring a metrics scope. A metrics scope lists all the projects and accounts that it monitors. A scoping project hosts the metrics scope. The scoping project stores the alerting policies and other configurations that you create for the metrics scope. The scoping project for a metrics scope is the project selected by the Google Cloud console project picker.
Alerting policies based on log-based metrics, like alerting policies based on other metrics, work across all projects in the metrics scope of the scoping project.
Log-based alerting policies don't operate on metrics scopes; the logs in projects are not part of a metrics scope. A log-based alerting policy evaluates only those logs that originate in the current project or that are routed to the current project.
For more information about metrics scopes, including multi-project metrics scopes, and about scoping projects, see the following:
Incidents and notifications
When the condition of an alerting policy is met, Monitoring opens an incident and sends notifications to the notification channels of the alerting policy. To see the details of the incident, click View incident in the notification message, or navigate directly to the Incidents page in Monitoring.
Alerting policies based on log-based metrics create incidents and notifications like all other metric-based alerting policies in Monitoring, as described in Alerting behavior. For more information about managing incidents for metric-based alerting policies, see Incidents for metric-based alerting policies.
Log-based alerting policies aren't metric-based alerting policies. When a log entry meets the condition of a log-based alerting policy, Monitoring creates incidents and notifications as follows:
The first time Cloud Logging writes a log entry that matches your alert query to a log bucket, an incident is created, and a notification is sent. If another matching log entry is then written, then a new incident is created only if the previous incident has been closed. However, it might take up to three minutes for a closed incident to be purged. When a matching log entry is received in the three minutes after you closed an incident, the system might reopen that incident rather than creating a new incident.
When you create a log-based alerting policy, you can specify the minimum time between notifications. For example, you select 10 minutes as the time between notifications. If the condition of your log-based alerting policy is met twice within that period, then you receive only one notification.
The maximum rate of notifications for log-based alerting policies is 1 notification every 5 minutes for each log-based alerting policy. However, if the query of your log-based alerting policy extracts label values, then each combination of extracted values represents its own notification timeline. For example, assume a log-based alerting policy extracts the values of a label. Assume that label can have two values. With this configuration, you could receive two notifications, one for each label value, in the same 5 minutes.
There is a limit of 20 notifications a day for each log-based alerting policy. If you reach this limit, then your notification includes a message that you have reached the limit for the day.
Incidents are automatically closed after 7 days, unless you configure a shorter period or close the incidents manually.
For more information about managing these incidents, see Managing incidents for log-based alerting policies.
Creating and managing alerting policies
You create, modify, and delete alerting policies based on log-based metrics in Cloud Monitoring, like any other metric-based alerting policy. For more information, see Managing policies.
You can create log-based alerting policies by using the Logs Explorer or the Cloud Monitoring API. You modify and delete log-based alerting policies in Monitoring. For more information, see Managing log-based alerting policies.
Viewing alerting policies
The Policies page in Monitoring lists all the alerting policies in your Google Cloud project. This list includes policies that use log-based metrics and log-based alerting policies.
-
In the Google Cloud console, go to the notifications Alerting page:
If you use the search bar to find this page, then select the result whose subheading is Monitoring.
- Select See all policies.
Log-based alerting policies appear in the list with the value Logs
in the
Type column. Alerting policies based on metrics, including log-based
metrics, appear in the list with the value Metrics
in the Type column.
The following screenshot shows an excerpt of a policy list:
Notification channels
You can send notifications from both metric- and log-based alerting policies to any of the notification channels supported by Monitoring. You must configure these channels before you can use them in alerting policies.
For more information, see Managing notification channels.
Authorization requirements
Using log-based metrics or log-based alerting policies requires authorization for both Cloud Logging and Cloud Monitoring.
For user-defined log-based metrics, see Permissions for log-based metrics.
For log-based alerting policies, see Permissions for log-based alerting policies.
Costs and limits
If you define your own log-based metrics, then the following apply:
- There are limits on the number and structure of user-defined log-based metrics. For more information about these limits, see limits for log-based metrics.
- You might incur charges for user-defined log-based metrics. For more information about costs associated with metric ingestion, see Chargeable metrics.
There are no charges associated with using alerting policies based on log-based metrics.
The following Monitoring limits related to alerting policies apply:
Category | Value | Policy type1 |
---|---|---|
Alerting policies (sum of metric and log) per metrics scope 2 | 500 | Metric, Log |
Conditions per alerting policy | 6 | Metric |
Maximum time period that a metric-absence condition evaluates3 |
1 day | Metric |
Maximum time period that a metric-threshold condition evaluates3 |
23 hours 30 minutes | Metric |
Maximum length of the filter used in a metric-threshold condition |
2,048 Unicode characters | Metric |
Maximum number of time series monitored by a forecast condition |
64 | Metric |
Minimum forecast window | 1 hour (3,600 seconds) | Metric |
Maximum forecast window | 2.5 days (216,000 seconds) | Metric |
Notification channels per alerting policy | 16 | Metric, Log |
Maximum rate of notifications4 | 1 notification every 5 minutes for each log-based alerting policy | Log |
Maximum number of notifications | 20 notifications a day for each log-based alerting policy | Log |
Maximum number of simultaneously open incidents per alerting policy |
1,000 | Metric |
Period after which an incident with no new data is automatically closed |
7 days | Metric |
Maximum duration of an incident if not manually closed | 7 days | Log |
Retention of closed incidents | 13 months | Not applicable |
Retention of open incidents | Indefinite | Not applicable |
Notification channels per metrics scope | 4,000 | Not applicable |
Maximum number of alerting policies per snooze | 16 | Metric, Log |
Retention of a snooze | 13 months | Not applicable |
2Apigee and Apigee hybrid are deeply integrated with Cloud Monitoring. The alerting limit for all Apigee subscription levels—Standard, Enterprise, and Enterprise Plus—is the same as for Cloud Monitoring: 500 per metrics scope .
3The maximum time period that a condition evaluates is the sum of the alignment period and the duration window values. For example, if the alignment period is set to 15 hours, and the duration window is set 15 hours, then 30 hours of data is required to evaluate the condition.
4If the query of your log-based alerting policy extracts label values, then each combination of extracted values represents its own notification timeline. For example, assume a log-based alerting policy extracts the values of a label. Assume that label can have two values. With this configuration, you could receive two notifications, one for each label value, in the same 5 minutes.