Utility Data Collection on AWS Lambda · Integration & Automation
Evantech
Evantech is a United States company whose platform collects account and usage data from utility providers' customer portals. We added new providers to its scraping system, made the automations reliable on AWS Lambda, and handled the CAPTCHA that was blocking a key data export.
Built with
- Python
- Selenium
- BeautifulSoup
- AWS Lambda
- Docker
- AWS CloudWatch, 2Captcha
Expertise
- Browser automation
- Headless Chrome on AWS Lambda
- Provider adapters for an existing architecture
- CAPTCHA handling
- Log-driven debugging
- Credential security review
Verticals
- Energy and utilities
- Data platforms
- SaaS
Evantech's platform signs in to utility customer portals, such as Southern California Edison, Duke Energy and Salt River Project, and downloads account data on a customer's behalf. Each provider needs its own automation, and the roadmap called for around twenty of them. Evantech had a reusable Python and Selenium codebase, but every new provider still had to be mapped, built and tested against a live portal.
There was a harder problem underneath. The scrapers worked on developers' machines but kept failing once deployed to AWS Lambda, where the headless browser crashed mid-run with renderer disconnection errors. The team had spent time on it without finding the cause, and with several providers waiting, they needed someone who understood both browser automation and how it behaves inside Lambda.
We started by learning Evantech's architecture rather than writing around it. A job created in the platform selects a provider, a shared login module signs in, and the authenticated browser is handed to that provider's scraper through a single entry point. We built the Southern California Edison adapter to plug into exactly that pattern, following the structure of the existing Duke Energy scraper, and returned results in the JSON format the platform's API expects.
The Edison portal put a CAPTCHA in front of the date-range CSV export, which is the data Evantech needed most. Rather than drop that part of the scrape, we recommended a CAPTCHA-solving service we had used successfully before, integrated it, and the export ran reliably from then on.
For the Lambda failures, we treated it as an environment problem, not a code-logic one. Pages render more slowly inside Lambda's container than on a laptop, so scripts that clicked as soon as an element existed were racing the renderer. We worked through the CloudWatch logs for the Duke Energy and Salt River Project scrapers, traced the Salt River Project failures to its login and authentication step, and reworked both scrapers to wait for the page states they depend on so they could be deployed to Evantech's system.
Along the way we flagged a security issue in the codebase: login credentials for several platforms were hardcoded in the scripts. We recommended moving them out of the code and into managed configuration, and raised it with Evantech's lead developer on a call.
Southern California Edison joined Evantech's list of supported providers within a week, including the CAPTCHA-protected usage export, built to the same pattern as the rest of the codebase so the team can maintain it alongside the others.
The Duke Energy and Salt River Project scrapers were fixed and deployed to Evantech's AWS Lambda setup, giving the team a working approach for the remaining providers on their roadmap.
Evantech kept us on the engagement through December 2024 and closed it with a five-star review.
Client feedback
What the client said
Exceptional Python Developer – VK Delivers Beyond Expectations!
Working with VK has been an absolute pleasure and a game-changer for our projects. Their mastery of Python is nothing short of extraordinary. VK consistently delivers clean, efficient, and highly scalable code, whether it’s building complex scripts from scratch or optimizing existing processes.
VK’s problem-solving skills are unparalleled—they approach challenges with creativity and precision, ensuring that every solution is robust and future-proof. They’re not just a coder; they’re a collaborator who genuinely understands our goals and works tirelessly to achieve them. Communication with VK is seamless; they keep us informed at every stage and are always open to feedback, which they implement flawlessly.
Thanks to VK’s dedication and expertise, we’ve been able to achieve milestones faster than anticipated, with exceptional quality. They’ve truly set the standard for what a top-tier developer should be. If you’re looking for a Python developer who’s passionate, skilled, and dependable, VK is the one you want on your team!
We couldn’t be happier and look forward to working on many more projects together. VK is, without a doubt, a 5-star professional!
Talk to an engineer
Like what you see?
Tell us about the system you need. We reply with questions and a suggested first step.