Why does cloud traffic monitoring need optimizing?
The simple fact is that cloud infrastructure providers charge by the amount of traffic transferred, and it makes no difference if it’s your actual data or monitoring metadata. At the same time, you still want to maintain a certain level of visibility into your traffic because by losing visibility, you lose control over your environment.
To prevent your cloud infrastructure costs from ballooning due to monitoring, you want to achieve a balance where you keep the overall amount of traffic for monitoring purposes to a minimum without compromising on visibility.
What options does Flowmon offer?
Flowmon 12 improves and expands flow log analysis support across the primary public cloud services. Flow logs are a form of network telemetry available natively in public cloud environments. Once configured, system administrators can use Flowmon to analyze flow logs to provide visibility into cloud traffic. This can be done across multiple cloud providers and incorporate on-premises infrastructure to support hybrid multi-cloud monitoring for much less cost than using separate native monitoring options.
Figure 1 below shows the overall concept of a hybrid deployment with a Flowmon Collector as a central aggregator gathering metadata from Probes (in this case both hardware and cloud, although only cloud ones will feature in the later discussion) and native flow logs.
Figure 1 – A sample hybrid deployment utilizing native flow logs, cloud traffic mirroring, and on-prem traffic
The Flowmon Collector can be both a hardware, virtual or cloud appliance, and standard sizing rules apply. A hardware Collector or virtual Collector deployed on-premise will reduce costs for computing resources in the cloud but will come with additional network egress costs. In the case of a cloud appliance, the situation gets reversed. Choosing one over the other will depend very much on the circumstances of your specific deployment.
Your deployment should include the Flowmon probes if you need:
- Highly reliable and accurate data for advanced network analytics and network performance metrics.
- In-depth visibility into network traffic, including application-layer visibility (L7 protocols).
- Advanced functionalities provided by Flowmon Application Performance Monitoring (APM) and Flowmon Packet Investigator (FPI).
However, operating Flowmon Probes in the cloud comes with the following caveats:
- You need either a cloud-native or 3rd-party vTAP/mirroring/brokering solution, which will introduce additional costs proportional to the volume of data passing through the monitored infrastructure.
- Running a Flowmon Probe in the cloud consumes computing resources and will incur extra costs.
- Depending on the cloud zones and regions, deployments may need multiple Probe appliances to service the monitored infrastructure.
In places where deep visibility and advanced features are not necessary, you can leverage native flow logs, which are very lightweight and consume much fewer resources.
Balancing resource consumption and visibility in Google Cloud
The following section focuses in more detail on the particularities of Google Cloud infrastructures. The cloud requirements for Azure and AWS will be very similar.
Figure 2 below shows a sample deployment with Google-specific terminology.
Figure 2 – A sample deployment with compute instances and cloud logging
In Google Cloud’s terminology, this is a Shared VPC deployment, where the cost of monitoring is impacted mainly by regionality or the deployment zone of resources. For this type of deployment, you will need the following services:
- vTAP/Mirroring – Google Cloud VPC Packet Mirroring
- Native Flow Logs – Google Cloud VPC Flow Logs
- with Google Cloud Logging and
- Google Cloud Pub/Sub
Packet mirroring and Probes
In a shared VPC deployment in Google Cloud, a copy of the network traffic from selected compute instances gets sent to an internal TCP/UDP load balancer and from there to one or more Flowmon Probes. You can use filters to mirror only parts of this traffic.
The mirrored traffic is counted toward Mirrored VM’s egress network traffic, and this needs to be considered during instance sizing. Your Probes must be deployed within the same region as Mirrored VMs (ideally in the same zone) to reduce these network egress charges.
Thus, the overall cost consists of:
- overprovisioning of egress network bandwidth for Mirrored VMs,
- internal TCP/UDP load balancer charges,
- cloud Compute charges for Flowmon Probe instance(s),
- network egress charges for mirrored data,
- network egress charges for flow data.
Flow logs
Flow logs are generated in subnets and published by logging to the pub/sub messaging infrastructure. Flowmon Collector appliances are subscribed to that infrastructure to retrieve and process the flow data. Hence their overall cost consists of:
- flow logs generation charges,
- logging charges,
- Pub/Sub charges,
- network egress charges.
Collectors
In Google Cloud terminology, Flowmon Collector appliances are cloud compute instances. The cost of standalone appliances consists of:
- Cloud Compute charges,
- network egress charges.
Cost and infrastructure examples of Google Cloud monitoring
This example (Figure 3) consists of three subnets – a DMZ for access to external networks, an application subnet to host the website, and an application database subnet with an internal load balancer sitting between the two.
Figure 3: Example Infrastructure
The assumption here is that we expect 1Gbps throughput to and from external networks, 5Gbps in communication between the two application subnets, and 10 Gbps between communicating instances in the database subnet. At the same time, we want:
- Basic visibility into application internals
- In-depth visibility at the perimeter
- A reasonable costs vs. benefits ratio
The options are to use either only Flowmon Probe and Collector appliances, only Google Cloud VPC flow logs and Collector appliances, or a mix of both. NOTE: All costs shown below are for illustrative purposes only and are subject to change by Google Cloud at any time. However, the costs do show the scale of the difference between the options.
Option 1
If you choose to go with the first option and deploy a Probe on every subnet (Figure 4), you could estimate your monthly Cloud Compute costs at $2,868 with total monthly network costs for traffic mirroring $51,840 (assuming minimal cross-zone traffic). Indeed, you would get perfect visibility, but the monitoring costs are astronomical.
Note that you can reduce the mirroring costs by applying filters and only mirroring selected protocols or hosts, but this naturally comes with a loss of visibility plus increased management overhead.
Figure 4: Example Infrastructure - all Subnets monitored by Flowmon Probe
Option 2
If you monitor all your subnets using only flow logs and send them to your central Collector, your monthly Cloud Compute cost could be estimated at $1,448, and monthly flow logs cost $2,726.
Your network egress costs would be negligible, and your total costs would be highly favorable, but with very low visibility – especially on the perimeter, which is risky, to say the least.
Figure 5: Example Infrastructure - all Subnets monitored by flow logs
Option 3
This option combines the best of both worlds, using Flowmon Probe appliances with packet mirroring on the perimeter to provide complete visibility into incoming and outgoing traffic, complemented by flow logs for low-granularity visibility into the internal network between tiers.
With packet mirroring and a Probe on your DMZ subnet, flow logs in your two internal subnets, cloud logging, pub/sub messaging, and a virtual Collector, Cloud Compute costs can be estimated at $1,800, network costs for mirroring at $3,240 and flow logs cost at $870 per month.
In this way, you achieve an optimum balance between visibility and cost, reserving traffic mirroring for only where it is needed and covering the rest with basic visibility.
Figure 6: Example Infrastructure - Combination of Flowmon Probes & flow logs
Comparison of Google Cloud monitoring costs for different deployment methods
| Deployment method | Google Cloud costs | Level of visibility | |
|---|---|---|---|
| Option 1 | Only Probes | $54,708 | Excellent |
| Option 2 | Only Flow Logs | $4,174 | Basic |
| Option 3 | Probes & Flow Logs | $5,910 | Reasonable |
This table shows that Option 3 provides a reasonable compromise in monitoring visibility at just under 11% of the cost of Option 1. This trade-off in visibility versus cost will appeal to many organizations. Remember that you can deploy additional Flowmon Probes to monitor mission-critical applications as required. This will increase the costs incrementally. How much of an increase will depend on how many additional Probes get deployed.
Conclusion
The additions and enhancements to flow log monitoring in Flowmon 12 deliver more cost-effective options for all organizations using AWS, Azure, and Google Cloud. The scenarios and figures we have presented here are focused on Google Cloud, but deployments for AWS and Azure will be similar. Visit the Flowmon 12 page to find out more details, or contact your Progress Flowmon Partner Account Manager or sales representative to find out more.
FAQs
Are VPC flow logs expensive? ›
VPC flow logs cost $0.50 per GB for the first 10 TB. For 850 GB this is $425.00. In regards to what should you do with the logs, analyze them. They are your log files.
What are the benefits of VPC flow logs? ›You can use VPC Flow Logs as a centralized, single source of information to monitor different network aspects of your VPC. VPC Flow logging gives security engineers a history of high-level network traffic flows within entire VPCs, subnets, or specific network interfaces (ENIs).
What kind of data can a company collect with VPC flow logs? ›VPC Flow Logs records a sample of network flows sent from and received by VM instances, including instances used as GKE nodes. These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.
How to implement VPC flow logs? ›- Open the Amazon VPC console.
- In the navigation pane, select Your VPCs or Subnets.
- Select one or more VPCs or subnets whose traffic you need to log.
- Select Actions > Create flow log.
- Specify the type of IP traffic to log in Filter.
Sudden increases in CloudWatch Logs bills often result from an increase in ingested or storage data in a particular log group. Use CloudWatch Logs metrics to check data usage, and review your AWS bill to identify the log group that's responsible for bill increases.
Why are CloudWatch metrics so expensive? ›From the graph below I could see that most of the CloudWatch cost was attributed to DataProcessing-Bytes in the APS2 region. The cost increase is associate to a single Region. The next step was to identify which API operation was attributing to an increase in the data.
What are good uses of enabling flow logs? ›- Diagnosing overly restrictive security group rules.
- Monitoring the traffic that is reaching your instance.
- Determining the direction of the traffic to and from the network interfaces.
Amazon VPC Flow Logs provide visibility into VPC and instances network traffic. Flow records are small and have a fixed size, making them highly scalable, with longer retention times, even for large organizations. AWS CloudTrail provides the logs for monitoring the AWS Cloud environment itself.
Why are flow logs important? ›VPC Flow Logs is an AWS feature that tracks traffic between the network interfaces of your VPC. This feature plays a key role in ensuring your network resources are secure by enabling you to see what data in your VPC has been accessed and determine whether your firewalls are overly permissive.
How do I view flow logs in CloudWatch? ›To view flow log records published to CloudWatch Logs using the console. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/ . In the navigation pane, choose Logs, Log groups. Select the name of the log group that contains your flow logs to open its details page.
How do I get VPC flow logs to CloudWatch? ›
- In the navigation pane, choose Transit gateways.
- Select the check boxes for one or more transit gateways and choose Actions, Create flow log.
- For Destination, choose Send to CloudWatch Logs.
Open the Amazon VPC console at https://console.aws.amazon.com/vpc/ . In the navigation pane, choose Subnets. Select the checkbox for the subnet. Choose Flow Logs.
How to read VPC flow logs in S3 bucket? ›To view flow log records published to Amazon S3
Open the Amazon S3 console at https://console.aws.amazon.com/s3/ . Select the name of the bucket to open its details page. Navigate to the folder with the log files. For example, prefix /AWSLogs/ account_id /vpcflowlogs/ region / year / month / day /.
- Clean Up. Terminate EC2 instance. Delete the flow log from the subnet. Empty the bucket. ...
- Summary. Remember to add both the bucket and the objects within the bucket as resources within the policy.
VPC Flow Logs can be created at the VPC, subnet, and network interface levels.
How do I reduce cost on CloudWatch? ›CloudWatch dashboards
To reduce costs, delete unnecessary dashboards. If you're using the AWS Free Tier, keep your total number of dashboards to three or less. Also be sure to keep the total number of metrics across all dashboards to less than 50.
- Understand what services and resources are covered by the AWS Free Tier.
- Monitor Free Tier usage with AWS Budgets.
- Monitor costs in the Billing and Cost Management console.
- Find and terminate resources when you're done using them.
- Shut down any unused EC2 instances you have. ...
- Minimize oversized instances and volumes. ...
- Use private IPs. ...
- Take a snapshot and delete low-used Amazon EBS volumes. ...
- Plan for AWS Savings. ...
- Utilize Reserved Instances (RI) to reduce RDS, Redshift, ElastiCache, and Elasticsearch expenses.
While Amazon CloudWatch provides excellent monitoring and management capabilities for AWS resources, it has limited support for monitoring resources outside of AWS. This can be a limitation if you have a hybrid infrastructure or use third-party services.
How do I see CloudWatch cost breakdown? ›Before we can start with our cost analysis, we need to gather some information about our current CloudWatch costs. In your AWS account, go to AWS Cost Explorer, hit Explore Costs, filter Service by CloudWatch in the filter panel on the right-hand side, and group by usage type on top of the chart.
What metrics can be monitored through CloudWatch without being charged? ›
CloudWatch Free Tier includes 10 Custom Metrics and 10 Alarms, 1,000,000 API Requests, 5GB of Log Data Ingestion and 5GB of Log Data Archive; 3 Dashboards with up to 50 Metrics Each per Month.
What are VCN flow logs? ›VCN flow logs shows details about traffic that passes through your VCN. VCN flow logs help you audit traffic and troubleshoot your security lists. Flow logs are enabled and managed using the Logging service. For more information, see Logging Overview. Flow logs can be generated from VNICs, PEs, and RCEs.
What is flow logs in Azure? ›Network security groups flow logging is a feature of Azure Network Watcher that allows you to log information about IP traffic flowing through a network security group.
Should flow logs be enabled for every network security group? ›The Flow logs should be configured for every network security group policy audits all existing network security groups in a scope by checking all Azure Resource Manager objects of type Microsoft. Network/networkSecurityGroups .
What is difference between CloudWatch and CloudWatch logs? ›CloudWatch delivers metric data in 5 minutes periods for basic monitoring and 1 minute periods for detailed monitoring. The CloudWatch Logs Agent will send log data every five seconds by default.
What is the difference between CloudWatch and flow logs? ›CloudWatch monitors performance and CloudTrail monitors activities. On the other hand, VPC Flow Logs are responsible for recording the network traffic. For instance, it will show you if a connection from a computer to your EC2 instance was accepted or denied.
What is the difference between CloudWatch and CloudTrail logs? ›The two services, Amazon CloudWatch and CloudTrail can be used together. CloudWatch focuses on the activity of AWS services and resources, reporting on their health and performance. On the other hand, CloudTrail is a log of all actions that have taken place inside your AWS environment.
How do you check flow logs? ›Open the Amazon VPC console. In the navigation pane, select Your VPCs or Subnets. Select the VPC or subnet whose logs you need to view, then click the Flow Logs tab.
What is the primary purpose of log monitoring? ›The primary purpose of log monitoring is to detect unauthorized access attempts and other malicious activity. However, log data can also be useful for troubleshooting network problems or identifying potential security vulnerabilities.
Why do we use nacl with VPC? ›A network access control list (NACL) is an optional layer of security for your VPC that acts as a firewall for controlling traffic in and out of one or more subnets. You might set up network ACLs with rules similar to your security groups in order to add an additional layer of security to your VPC.
What is the best way to search CloudWatch logs? ›
To search your logs using the console
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/ . In the navigation pane, choose Log groups. For Log Groups, choose the name of the log group containing the log stream to search. For Log Streams, choose the name of the log stream to search.
- Create a custom ec2 IAM role with Cloudwatch log write access.
- Install Cloudwatch logs ec2 agent.
- Configure log sources in the Cloudwatch agent configuration file.
- Start the agent with the configuration file.
- Validate logs in the Cloudwatch dashboard.
When CloudTrail logging is turned on, CloudTrail captures API calls in your account and delivers the log files to the Amazon S3 bucket that you specify.
What is the retention period of flow logs? ›A flow log data retention period of 90 days or more, should allow you to collect the necessary amount of logging data required to check for anomalies and provide details about any potential security breach.
How do I trigger Lambda from CloudWatch logs? ›- Log Group. Select the log group that serves as the event source. Events sent to the log source will trigger your Lambda function.
- Filter Name. Enter a filter name.
- Filter Pattern. May be left blank. ...
- Enable trigger—Check the box to enable the trigger immediately.
- Sign in with sufficient permissions as documented in Step 2: Set up access permissions.
- In the navigation pane, choose Log groups.
- On the Log Groups screen, choose the name of the log group.
- Choose Actions, Export data to Amazon S3.
VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to Amazon CloudWatch Logs and Amazon S3. After you've created a flow log, you can retrieve and view its data in the chosen destination.
How will you monitor VPC network flow? ›You can use the following tools to monitor traffic or network access in your virtual private cloud (VPC). You can use VPC Flow Logs to capture detailed information about the traffic going to and from network interfaces in your VPCs. You can use IPAM to plan, track, and monitor IP addresses for your workloads.
How do I check my WAF logs on CloudWatch? ›- Open the Amazon CloudWatch console.
- In the navigation pane, choose Logs, and then choose Log Insights.
- For Select log group(s), choose one or more log groups to query that consist of AWS WAF access logs.
- (Optional) Choose a time range for the period that you want to query.
- Use query syntax to design queries.
VPC Flow Logs records a sample of network flows sent from and received by VM instances, including instances used as Google Kubernetes Engine nodes. These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.
How do I check my S3 bucket logs in CloudWatch? ›
- In the navigation pane, choose Logs.
- Select the name of the log group for your Lambda function (/aws/lambda/function-name).
- Select the name of log stream to view the data provided by the function for the instance that you launched.
- Go to Settings > Scheduler.
- In the left navigation menu, click Log Collection. ...
- Click Create Log Collection Job. ...
- Enter the name and description for the job. ...
- Select Sensor as the source for your new job.
- In the Action Type option, select Amazon Web Services.
- Prerequisites.
- Step 1: Get the 12 digit destination AWS account number. ...
- Step 2: Setup source S3 bucket. ...
- Step 3: Setup destination S3 bucket. ...
- Step 4: Attach policy to IAM user in destination AWS account. ...
- Step 5: Sync S3 objects to destination. ...
- Conclusion.
Each AWS account can create 100 buckets, and users can request a service limit increase to obtain more. The AWS account that creates a bucket owns it, and ownership isn't transferable. An S3 user can delete a bucket, but another AWS user can claim that globally unique name.
How many S3 buckets can you have with a new AWS account? ›By default, customers can provision up to 100 buckets per AWS account. However, you can increase your Amazon S3 bucket limit by visiting AWS Service Limits. An object can be 0 bytes to 5TB. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.
Which two services can be used to store VPC flow logs? ›VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to the following locations: Amazon CloudWatch Logs, Amazon S3, or Amazon Kinesis Data Firehose.
What is the sampling rate for VPC flow log? ›Flow sampling can be set from 0.0 (no sampling) to 1.0 (all logs). Default is 0.5 .
Do I need VPC flow logs? ›Flow logs can help you with a number of tasks, such as: Diagnosing overly restrictive security group rules. Monitoring the traffic that is reaching your instance. Determining the direction of the traffic to and from the network interfaces.
Does VPC cost money? ›Q. How will I be charged and billed for my use of Amazon VPC? There are no additional charges for creating and using the VPC itself. Usage charges for other Amazon Web Services, including Amazon EC2, still apply at published rates for those resources, including data transfer charges.
Why are logs so expensive? ›Log prices increase as demand for firewood grows
The data they gathered revealed that the price of kiln-dried logs has upscaled at a 'significantly greater rate than the overall rate of inflation which currently sits at 8.8 per cent for the 12 months up to September 2022.
How to configure AWS VPC flow logs? ›
- In the navigation pane, choose Transit gateways.
- Select the check boxes for one or more transit gateways and choose Actions, Create flow log.
- For Destination, choose Send to CloudWatch Logs.
Default VPC does not have a lot of the critical security features that standard VPC comes with, new resources should not be created in the default VPC and it should not be present in the Terraform.
What is the disadvantage of VPC? ›The Disadvantages of Virtual Private Cloud Works
VPC generally costs more to operate than standard cloud hosting. It can even be more costly than operating a private, on-premises cloud solution. So it's essential to weigh up the costs of moving data in and out of a VPC.
You can't have more than 255 gateway endpoints per VPC. This is the combined quota for the maximum number of interface endpoints and Gateway Load Balancer endpoints in a VPC.
Is VPC peering cheaper? ›Number of VPCs to be connected is lower (~<10). You need multiple VPCs' connectivity to On-premises. You want to minimize data transfer costs when significant volumes of data transfer across regions, VPC Peering is cost-effective.
What is a cheaper alternative to logs? ›Yes, briquettes are a fantastic fuel for a wood burning stove and are an alternative to high quality logs. Similar to logs, some briquettes can also be broken up to provide a kindling option when lighting a fire.
What is the cheapest way to buy logs? ›The most economical way to purchase kiln dried wood is by buying it in bulk from a local supplier. This is also the most sustainable option for reducing the distance to have the logs delivered.
How long does a 1 ton bag of logs last? ›A single bulk bag weighs 215kg and contains approximately 300 logs. If you use five logs a night in your wood burner, this would provide you with enough logs to last for around two months. That's plenty of fires to keep you warm and toasty!