Test Automation is becoming a hot topic in the Salesforce ecosystem. Automated testing increases test coverage beyond manual capabilities and allows users to adopt a scalable approach to “test smarter and release faster”.
To find out more, I spoke with Jonathon Wright, the Chief Technology Evangelist at Keysight Technologies. One area that Keysight focuses on is test automation – a great fit for the Salesforce ecosystem.
Jonathon is also President of Vivit Worldwide, which is the largest independent software user community. He’s been involved with helping individuals upscale as the demands for different tech skills change – a bit like our Salesforce Trailblazer community. Something that caught my attention was Jonathon’s LinkedIn profile URL, which is /automation. I think it’s safe to say that he is passionate about automation.
Chief Technology Evangelist: A Day in the Life
Lucy: ‘Chief Technology Evangelist’ is one of those job titles that is interpreted person to person. What does a ‘day in the life’ look like for you?
Jonathon: The normal kind of day – two screens, two simultaneous voice calls, going through tunnels, trying to communicate the vision of where our customers need to be.
“A lot of my day-to-day work is very customer-focused – we’re all about listening to what our customers really need.”
I’m the face for that – evangelist piece around what is it, what we can do? What’s the art of the possible? But also, more importantly, how do we make sure we can deliver it for enterprises? Because every organization’s different.
Why Test Automation?
Lucy: Out of everything in your diverse interests and career, why test automation? What drew you to that?
Jonathon: I started off in the ‘90s when I was like most people – a young graduate, and started working for an engineering company. On my first day, they gave me this big pile of test scripts to run. And it was me picking phones up on these voiceover IP phones, making phone calls, hanging up at reception. And it was for 30 days!
I was enthusiastic. I turned up to my boss with this whole stack of test cases, which was about a hundred pages with a little tick next to each one. I said: I’ve tested all of them, honest! And he said: I completely believe you because it’s run perfectly for the last 10 years, but we still have to do it every six months.
I was so gutted, thinking: I’ve got to do this again in a couple of months’ time. It’s just soul-destroying.
That was my first taste of test automation – they actually brought in a machine which was called an Abacus – not like a counter Abacus – but the machine was very similar to some of our equipment at Keysight (instrumentation). It just plugs directly into hardware, which is kind of where Hewlett-Packard (HP) started, and most of Silicon Valley, in fact. They were building this instrumentation and Keysight is actually originally HP.
It’s a really good point to know – they started this automation journey and they automated all these phone calls. I was just like, wow – that’s a lot easier.
“What it does is, it frees us to actually do the more important things – the subject matter experience, the domain expertise we have, understanding our business, and then getting Salesforce to work just how we need it for the best efficiency.”
That’s really the way of automation and, within the industry, as you pointed out with my URL, they refer to me as the Automation Cyborg, sent back in time to “save the world from bad software”, which is actually Keysight’s (or Eggplant’s) old motto! So, it’s really good to be back here, helping people save time, and do things faster and better.
Lucy: And ensuring everybody has the best quality of life as well. Those soul-destroying IT jobs that you’ve taken on, but didn’t know any different at the time – I’m sure many people can relate to that.
Challenges of Adopting Test Automation
Lucy: For anyone who hasn’t explored test automation, what would be the top three challenges of adopting a test automation routine or culture within an organization – within a managed packaged software – like Salesforce?
1. Maintaining User Access and Permissions
Jonathon: One of the things I love about Salesforce is that there are a lot of managed service providers who are subject matter experts, who can help you make sure you implement Salesforce correctly.
Not locking down things that people need, filters, or potentially objects that they need for their day-to-day practice. User roles and responsibilities are really difficult. I think the other aspect to adopting something like automation is to understand the blueprint (schematic) of how you’ve implemented Salesforce.
“This is where one of our solutions (the Salesforce Accelerator) comes in. We look at the unique ‘DNA’ of your Salesforce instance and we create a ‘digital twin.'”
So, if you think about Apollo 13 and Tom Hanks stuck in space, they’ve got a particular problem with the system, and they use a digital twin to represent it back in Houston (the problem) and try lots of different combinations to fix it.
This is the same approach. We’re able to take a digital twin, a snapshot of your system, then we’re able to run tests against that digital twin, against your actual Salesforce sandbox environment or your pre-production environment. It takes a lot of the problems with lots of different environments, all slightly configured in a different way. Because the digital twin is at the heart of your organization, it helps you test the system.
2. Maintaining Integrations to Other Platforms
We all know there’s more involved than just Salesforce – it’s actually the ecosystem that surrounds it. So you might be reliant on Big Machines, SAP or other third-party integrations.
“That’s also really difficult to test because you’ve got to test somebody else’s system – which is out of your control – and how they integrate with it.”
Those kinds of integrations with Salesforce make it hugely complicated.
That’s why we created something called “true E2E” – the ability to test from “ecosystems to ecosystems”, your entire IT stack – no longer testing in silos (one vendor at a time) like was traditionally the case. We understand the complexities of modern-day hybrid cloud architecture can touch hundreds even thousands of upstream and downstream endpoints.
3. Testing User Experiences Across Devices
Through your mobile journeys – you might be using a mobile device to do two-factor authentication, connecting to an upstream system, or logging back in as a different user who’s approving a process or a workflow. We test that whole process.
“I think that’s the real challenge – how do you make it realistic to how your users use Salesforce?”
Lucy: I heard a statistic the other day – the average company is using around 900 apps. Salesforce Admins are so specialized with Salesforce, and then there’s always that concern about how the integration is being affected, so it puts their mind at rest.
Who is Keysight?
Lucy: Who is Keysight (Eggplant)? And who are you really supporting in an organization?
Jonathon: You could look at it from an IT infrastructure point of view – an IT department who delivers your Salesforce changes. I generally think it’s important to get in contact with the business and understand what they’re doing.
“We use the phrase ‘we eat our own aubergine’ (eggplant in the US!) – we use this tool to test our own Salesforce instance and our internal teams are using it. We try to do this for ourselves first and then learn from that.”
Then we start working with partners who are strategically Salesforce implementation partners. We work out what we need to learn – what are the use cases the client organization has?
We’re exploring the ‘shift right’ approach, which is where we turn on telemetry in the production environment (which is what we’ve got in our Salesforce org right now). Our sales team are all using Salesforce.
“We’re capturing all the user flows and interactions they’re doing with the system. We can take this behavioral data, then create and overlay a digital twin – which will show all those possible permutations. Essentially, we’re saying is: This is what your actual users are doing.”
Not only that, we can see where they potentially drop off, they get to a point where they can’t run a report, or they can’t filter in the right kind of way.
I was on a call recently with our research and development team, who were saying certain lists don’t function the same when in French terminology. But we’ve detected this issue, and many others, from testing all the permutations of every dropdown box, every single form, every workflow etc. And that’s millions, if not billions of combinations, which is a huge amount of time saved, much better than me being locked in a lab for 30 days!
The Future of Test Automation
Lucy: You once used an analogy of a Rubik’s cube to describe those limitless combinations of a users’ next steps. So overall – you’ve hinted to this already – which direction do you see test automation going in the future?
Jonathon: On the one hand, we’re not aiming to replace anybody – it’s not going to be a robot taking your job.
What are the pain points, and what have we got at the moment? And as you pointed out, things like regression, like patching, like changes from your third-party integrations. These are business-critical systems so each time a change is made, you need to be sure that everything is fine. For example, you just changed the currency from dollar to yen, you have to make sure all the reports remain useable and the data can be extracted, transformed and loaded (ETL), so that the other integrated systems can function when receiving the data.
Even if you’re just changing one small thing, the complexity can be huge. Why make it so person-intensive when you can automate it? There are advantages of using automation and that’s to free up those subject matter experts to be doing better things, but also to be validating that the solutions are working as they should in Salesforce.
Test Automation Myths to Bust
Lucy: Test automation, to a layman, doesn’t really mean a lot. But then, people can start making up their own ideas about what it means. What are some test automation myths that you’d like to bust?
Jonathon: I’ve heard some fantastic myths, like: You’ll never find another bug once you’ve found it once, because it’s only going to go through the same route. We’ve got a bug-exploring algorithm, which will go off and say: I found this list box, which wouldn’t allow you to input French in when it requires the particular ISO standard. Every other list box could still be subjected to the same problem, so we’ll go and test those.
Some of the statements that we think are myths of automation, are actually true – that it’s hard to implement and takes a lot of time.
“We’ve got this great approach of ‘shifting left’ and creating that digital twin, really quickly. If you turn it on within your sandbox environment and do the digital twin approach, you can generate all kinds of insight based on real user behavior. This is how we plan to make test automation easier and lower the barrier of entry.”
Lucy: It’s amazing that the DevOps conversation has absolutely exploded in the Salesforce ecosystem since 2020/2021. I see the technology you’re referring to as the next generation of test automation that we’re going to need in Salesforce projects.
Jonathon: I refer to DevOps as ‘Ops Dev’, because it’s driven by operational excellence – what your business actually does.
“The idea with DevOps is to blur the boundaries between development, wanting to release things faster to the business. And then the business actually needing to support that from an SLA perspective and an instant management position. Part of it is continuous feedback.”
It’s case of make, check (as in test), and then learn. That feedback loop can be much more rapid. And that’s where we’re trying to release faster, add more value to your customers, and give them access to the latest and greatest technologies.
Test Automation and Salesforce
Next I spoke with Ethan Chung, Engineering Manager at Keysight, about how the company is emerging as a Salesforce solution.
Lucy: You work with a lot of customers. What desires, and challenges, are you seeing from those customers, when using Salesforce specifically?
Ethan: One of the biggest challenges comes from the fact that Salesforce is not just the CRM. There are so many additions from the AppExchange, and the partner companies making their own applications. Everything combined, makes it incredibly complex. Adding business logic on top of that, is effectively making an environment that is more and more tricky to manage day in and day out.
This is why Salesforce Administrators exist. They have to have an intimate knowledge within their Salesforce environment, often able to track every rule and automation that goes across it.
Oftentimes these rules and automations can conflict with each other – because there are multiple managers or multiple different sales admins working across the system – or anytime they have a new add-on.
So, the big challenge comes from one simple question: How do we make sure every single one of these are working in sync? This complex system is really where Eggplant’s been solving this problem.
With Eggplant, we focus very much on a modeling approach.
A lot of our customers come to us starting from very linear journeys where you have problem segments – where you go, I want to check the user can go through A, B, and C – very simple journeys. However, it’s incredibly difficult to do an exhaustive review of your environment for true testing needs.
We’ve been helping our customers in converting Salesforce into a model-based approach. Once we know what Salesforce looks like from your perspective – we map that journey out into a model, which we call the digital twin within Eggplant. And with that, you can intelligently move across the application like a user would do.
“Think of it like journeys across a map – go left, left, right, right – then the final destination. That’s the standard model that testing would do within Salesforce and any other application. What we have effectively done is the same as Google Maps would do. We map out every single road – we map out every journey possible.”
Instead of making a test where you say: Oh, let’s make a happy path and go from A to B to turning left and right to the correct locations. You just say: I need to go from A to B and then have it automatically generate all different path scenarios. I think that’s really been our biggest impact for customers with Salesforce – reducing that complexity so they no longer need to create every single path themselves, which is a testing nightmare.
Lucy: Clicks not code is a blessing and a curse. Everybody somewhat technical, with the correct permissions could be doing their own thing – there needs to be some governance. I see testing proactively brings a whole level of governance so that everybody’s taking ownership for that full user journey.
Ethan: Exactly. We know that Eggplant already works across Salesforce and we’ve been modeling for a long time with Salesforce challenges and it’s actually where we are moving into in the future.
The biggest challenge now is building the map. Historically, we’ve been building the map with the customers – the customers using our solution to build the digital twin.
Where we’re moving towards Salesforce (and other solutions) is how do you automatically generate this map itself, off Salesforce? That’s always going to be the biggest time sap across any modeling approach. With machine learning, ultimately, it’s only as good as the data set and the underlying data structure itself.
“Where we’re moving now with our Salesforce solution, is effectively to ‘one press of a button’. We map out the Salesforce environment. We know what it looks like. We know what possible paths there are.”
Final Thoughts
Thank you to Jonathon and Ethan for sharing their invaluable insights about test automation and how it relates to Salesforce professionals.
- Challenges of adopting test automation include maintaining user access and permissions, maintaining integrations to other platforms, and testing user experiences across devices.
- Adopting test automation effectively is to understand the blueprint (schematic) of how you’ve implemented Salesforce. Having technology that can look at the unique “DNA” of your Salesforce instance can generate a “digital twin” to run more thorough tests.
- Users don’t follow linear journeys predictably through A, B, and C. It’s incredibly difficult to do an exhaustive review of your environment for true testing needs. What Keysight have effectively done is the same as Google Maps would do. They map out every single road – they map out every journey possible.
About Keysight:
Keysight Technologies, Inc. (NYSE: KEYS) delivers advanced design and validation solutions that help accelerate innovation to connect and secure the world. Keysight’s dedication to speed and precision extends to software-driven insights and analytics that bring tomorrow’s technology products to market faster across the development life cycle.
Eggplant test and automation intelligence solutions deliver continuous testing of applications, devices, and digital services enabling your teams to deliver higher quality applications faster than ever before.
