As the demand for larger-scale testing grows, LoadGen's capability to seamlessly handle up to 1500 users has set a high standard. However, the importance of robust testing infrastructure cannot be overstated, especially as the load on the Director escalates, making CPU optimization crucial. In the Q1 2024 release, LoadGen is poised to make a substantial leap forward. This blog delves into our groundbreaking test involving 10,000 test users, showcasing LoadGen's commitment to meeting the evolving needs of performance testing.
LoadGen Load and Performance Testing on-premises components
For the massive load test scalability update in LoadGen Load and Performance Testing, the architecture comprises several key components designed to work in synergy. These include:
The LoadGen Director for setting up load profiles and managing test users, scenarios, and environments;
The LoadGen Studio for building simulation workloads;
The LoadGen Analyzer for in-depth analysis during and after tests, using charts to combine user action transactions with performance metrics;
And the LoadGen Agent, which acts as distributed bots/launchers to launch various sessions. Together, these components enable LoadGen to effectively manage and scale up to testing 10,000 users, demonstrating a significant advancement in testing capabilities.
LoadGen architecture
The architecture of LoadGen is structured running all sessions (both VDI and WebTesting) concurrently in separate processes. LoadGen Agents initiate these sessions and collect results, including application starts, measurement messages, and internal statuses. They then batch these results and communicate them to the LoadGen Director.
In the architecture of LoadGen, there are three main types of messages that facilitate communication and data management:
SMD (Session Messages Director): These messages have the highest priority and ensure the LoadGen Director receives critical session-related communications promptly.
DUAF (Data User Action Framework): This category includes messages related to user actions within applications or transactions, such as start, stop, and fail events.
PM (Performance Metrics): These messages contain performance data from the systems under test, including various performance counters.
This structure allows LoadGen to efficiently manage and analyze the vast amounts of data generated during testing, ensuring high levels of performance and insight.
Infrastructure
For the massive load test scalability update, the infrastructure deployed in Azure is tailored to optimize the performance and reliability of LoadGen components during stress tests. The setup includes:
LoadGen Director on a Windows Server 2022, utilizing Azure's Standard_D32ds_v5, which offers 32 vCPUs, 128 GiB of memory, and 1200 GiB of SSD storage, designed to manage high-load scenarios efficiently.
LoadGen Agents, 20 instances running on Windows 11 Pro, are powered by Standard_D8s_v3 VMs, each with 8 vCPUs, 32 GiB of memory, and 200 GiB of SSD storage, enabling simultaneous session launches and result collection.
SQL Server, hosted on Windows Server 2022, uses the Standard_D8ds_v5 specification, providing 8 vCPUs, 32 GiB of memory, and 300 GiB of SSD temporary storage, ensuring robust data handling and analysis capabilities. The choice of SQL Server on a Windows Server 2022 with specific Azure specifications has been questioned, with Azure SQL being a potential alternative for cost-effectiveness. This opens up a consideration that had not been previously contemplated. The idea of testing Azure SQL for performance and cost, including the benefits of shutting down services during idle times to save resources, has been suggested and could be an interesting comparison for future infrastructure planning and cost optimization.
This infrastructure supports LoadGen's advanced testing capabilities, allowing for extensive scalability and detailed performance metrics analysis.
The key elements in a massive test...
The key elements in enhancing LoadGen's scalability for massive load testing are the LoadGen Agents and the database. The LoadGen Director efficiently manages many sessions and data processing, but the LoadGen Agents or Launchers execute the core functionality. These agents handle the test user sessions, with each session running in its own process, requiring CPU, memory, disk, and GPU resources.
We conducted multiple tests with varying numbers of agents to determine optimal scalability: 8 LoadGen Agents for 5,000 users, 15 LoadGen Agents for 9,000 users, and 20 agents for 10,000 users. It's important to note that these tests serve as a guideline, and the results may not be directly transferable to all situations. Factors such as one's infrastructure, session types, and workload characteristics significantly influence the scalability and performance outcomes.
The database also plays a crucial role, in recommending a simple recovery model for efficient management and recovery. More details on recommended database models can be found in Microsoft's documentation on recovery models.
The test scenario
The test scenario involves a custom-developed LoadGen TestClient that mimics the functions of Citrix and FAT Client launchers, capable of performing various actions typical of a normal workload. It's aimed at stress-testing LoadGen components, rather than a target environment, with two primary objectives:
Directing 10,000 users within the specified time frame using the LoadGen Director.
Handling an extensive flow of messages from the individual Session launching processes - to the LoadGen Agents - to the LoadGen Director - and - to the SQL database.
The LoadGen Director initiates batches of 80 users every 10 seconds. The workload includes login actions, repeated tasks (bulkers), and idle time (coffee), which are cycled 20 times. Additionally, LoadGen Agents collect the most common performance counters (CPU, Memory, IOPS) every 5 seconds. The provided table outlines the expected and actual outcomes, as well as the duration of each transaction type in the testing process.
The workload
To understand the expectations and sequence of the user test, it's essential to consider the overall objectives and each specific action that a user undergoes. This typically begins with the initial logon, followed by a series of predefined tasks or transactions that mimic real-world usage, potentially including error handling and idle periods to simulate breaks in activity. The end-to-end process is designed to emulate typical user behavior in the system under test, with the aim of validating performance and stability at scale.
In the anticipated LoadGen test, each user session contributes to a substantial influx of messages to the database. With every transaction and subtransaction comprising a start and either a success or a failure message, the expected message count is significant. For example, during the ramp-up, 10,000 users generate two messages each for starting and stopping. In bulker operations, which include both successes and failures, a large volume of messages is generated due to repeated iterations. This consistent pattern ensures a thorough evaluation of the database's capacity to handle intense, concurrent data flows within the expected running times.
Transaction | Measurement | Type | Expected | Running Time |
Ramp-up | 10,000 test users | 80 users per 10sec | ~ 25 minutes | |
Logon | Logged on | Stop | 10,000 | 1 sec + ~ 1 sec |
Bulker | Success | Stop | 25*17*10,000 = 4,250,000 | ~ 100 sec per iteration/user |
Bulker | Fail | Failed | 25*17*10,000 = 4,250,000 | ~ 100 sec per iteration/user |
Coffee | Drinking Coffee | Stop | 1*17*10,000 = 170,000 | 5 sec per iteration/user |
Totale iterations | 17 x | ~ 207sec * 17 = 3519 sec |
Load scenario
LoadGen Insight
LoadGen Insight, poised for beta release in the second quarter of 2024, represents a substantial advancement in IT environment analysis. This service, accessible as a SaaS application, brings a suite of capabilities:
Comprehensive Environment Monitoring: An agent that oversees the entire IT infrastructure, detailing which user opens which application, including the time and version of the app.
Performance Analysis: Systematic tracking of system, network, and process performance (every 1 second! with minimal impact) alongside granular application resource consumption.
Resource Utilization Reporting: It captures intricate details like CPU, memory, and disk IO utilization, alongside network usage - every 1 second - doing so with negligible system impact.
Hardware and Update Audits: The agent not only reports on the machine's hardware specs but also on disk usage and Microsoft updates installation status.
Operational Control Tools: Includes RemoteView for hands-on control of workstations, and an upcoming RemoteExec feature for deploying PowerShell scripts and executables remotely.
During its testing phase, LoadGen Insight has proven effective in assessing its own system footprint and ensuring that the performance counters deliver accurate readings.
Results
In extensive testing for LoadGen Agent distribution, success peaked at handling 10,000 users with a remarkably low failure rate of 0.15%. The detailed testing spanned from 10:28 to 12:17, yielding millions of various counters across categories:
DUAF Counters: A total of 18,452,280, capturing detailed user actions and transaction states.
SMD Counters: Amounting to 748,875, prioritizing session messages for the Director.
PM Counters: Totaled 391,551, tracking performance metrics across the system.
The breakdown of the types of transactions includes a vast range of subtransaction names from connection initiation to transaction completion, with substantial counts for each category indicating robust activity and tracking precision. The variation in failure rates suggests that a greater number of agents correlates with a lower margin of error, underscoring the scalability and reliability of the LoadGen system.
Count | Type | SubtransactionName |
9985 | DenamikBeforeConnect |
|
9985 | DenamikInitializing |
|
9985 | DenamikPrepareConnection |
|
9985 | OnFlowStart |
|
9985 | SendLogonSuccessful |
|
9985 | SendMessageDirector |
|
9985 | SubTransactionStart | Logged on |
9985 | SubTransactionStop | Logged on |
4243625 | SubTransactionStart | Succes |
4243625 | SubTransactionStop | Succes |
4243625 | SubTransactionStart | Fail |
4243625 | SubTransactionFailed | Fail |
169745 | SubTransactionStart | Drinking coffee |
169745 | SubTransactionStop | Drinking coffee |
9985 | SendDUAFScriptSuccessful |
|
LoadGen Director end of the test
LoadGen Analyzer Transaction and SubTransaction overview
LoadGen Insight - System Performance - LoadGen Director machine
LoadGen Insight - System Performance - LoadGen Agent machine
LoadGen Insight - System Performance - Microsoft SQL machine
LoadGen Insight - Process Performance - 'LoadGen Director.exe'
LoadGen Insight - Process Performance - 'LoadGen LoadBot Agent.exe'
LoadGen Insight - Network Performance - LoadGen Director machine
LoadGen Insight - Network Performance - LoadGen Agent machine
LoadGen Insight - Network Performance - Microsoft SQL machine
Conclusion
The culmination of LoadGen's testing efforts has decisively demonstrated its enhanced capacity to manage large-scale testing environments. With the rigorous assessment of LoadGen's architecture, including the Director, Studio, Analyzer, and Agents, the system has proven capable of handling up to 10,000 users efficiently, with a minimal failure rate. This feat is a testament to LoadGen's commitment to scalability and precision in performance testing.