Problem
AI training loads are highly correlated batch operations. When loading or saving a chunk of training data, the GPU power rapidly decreases to idle temporarily before another rapid increase to full load. These fluctuations cause significant strain on the grid and have been shown to interact with inverter based generation control schemes.
High resolution waveform capture courtesy NERC whitepaper.
These fluctuations can last many hours or days as shown in this trend from an AI datacenter in New Mexico.
Hardware solutions
E-STATCOMs and Battery Energy Storage Systems (BESS)
Uses utility-scale super capacitor bank (or lithium iron phosphate LFP) behind conventional voltage source converter based STATCOM (or conventional inverter). This can virtually eliminate the load fluctuations measured at the grid level. See the green simulation trend overlaid on the NERC whitepaper trend.
Site Load Control with Load Banks
Maintains a set load by applying additional load when the GPU is loading the next chunk. I would say this technology is already being phased out due to other superior methods. The upside of this technology is that it uses off-the-shelf hardware and software that has existed for a while. But it is inherently lossy and is not able to supply power during load spikes so it has a limited ability to mitigate ramp-up issues. The SLC project in New Mexico was canceled before it got off the ground.
Timing diagram for Avtron site load control load banks.
Software solutions
Nvidia has developed software that can limit GPU performance during ramp-up and burn power during ramp down. Coupled with additional short-term battery storage, including special power supply units in the rack, this technique has been demonstrated to reduce sub-synchronous oscillations significantly. This graphic from Nvidia’s blog on the subject summarizes how the technology works.
This technology was tested in an Nvidia datacenter on their Megatron LLM where peak power demand was reduced by 30% and sub-synchronous oscillations on the AC system were reduced almost entirely.
Question for discussion:
Is anyone aware of any other technologies that can reduce the magnitude and severity of sub-synchronous oscillations on the grid? Does anyone have a good waveform capture I can “play-in” to our PSCAD or PSLF models to see if the oscillation is in danger of resonating with our system?