Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Posit Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
So today I am super excited to introduce Tom Grace and Darren Cope from HMRC. Let's start with Tom. Tom, I would love it if you could introduce yourself, tell us a little bit about your role, and maybe a little bit about HMRC in general.
Okay, hello. So yeah, I'm Tom. HMRC, for those who are unfamiliar or from outside of the UK, is the tax office. My role is as a tech lead on the Posit platform team. So what we do is we build the Posit installations. We have about seven or eight of those now of Workbench and Connect. We build those, and we make sure that they are shaped the right way so that the data scientists across the organisation can do the best work they can. The tech lead thing is it's partly technical, but it's also quite heavy on the culture aspect of leadership. So it's making sure that our team is a good team to be on, and also making sure that we are approachable and can be, and find it easy to be empathetic with those who are using the platforms and systems that we build to win.
I think that the coolest thing that you said in that whole thing was that you have a Posit platform team, which means you have a team of people dedicated to making sure that people can use the tools that you have and know how to use them. I think that that's really rare and kind of amazing because I can hear from people in the chat, I'm sure all the time, that we get tools from vendors, from suppliers, and then they kind of come to our organization and die out. Nobody knows how to use them. They're not very well supported. It really takes an internal effort to make sure that that happens.
Oh my God, it works. I am on my phone. How thoroughly modern. All the devices fail me.
I was going to say that would be fantastic because I know that Darren is a Posit subject matter expert, which is a very unique role. I'm not sure I know any other Posit subject matter experts at other organisations. Tom, what is Darren's role on the team?
So Darren is part of, he sits between the platform team and also our fairly new Posit adoption team. And it's his job to understand the nuts and bolts of what it is people need to do on the platform. So the platform team themselves are coming at it from the perspective of, oh, we've got some servers over here. And Darren's coming at it from the perspective of I've got some data sets. How do I get access to those? What do I want to do with them? What kind of problems are coming down the track? How can we use this better? Are there situations where people are using legacy tooling at the moment where with a bit of support, they could move across to something that's a bit more modern and easier to work with? So it's a greater focus on user need rather than the technical aspect of it.
One thing I heard from Darren was that when I talked to him earlier this week, he said, I just came out of a coffee and coding session for people who want to use Posit tools and have never used them before. And he had over 160 people in there, which is wild. That's as many people as we have on the hangout call right now. So all those people from your organization who are interested in using different tools, who get the chance to have an empathetic person sitting there and saying, like, let me just walk you through how this works.
Yes. I mean, it's a very large organization. We pretty much we've got a whole number of everything users. There's a fair number of SAS users. There's, as you'd expect, shed loads of Excel users. But probably pretty much any large enterprise tool that you can think of, there's a fair few users of that. There are people who are perhaps multi-decade SAS users who would be interested in moving across. But it's quite a learning curve for them.
Tools, data, and scale at HMRC
So yeah, I don't do any data analysis myself, but it's the organization covers everything from things like import and export duty. So there are teams who are doing things like running analysis on bulk corporate returns, or they are looking back at the historical data, looking for patterns. There are teams who are modeling proposals for the government around, you know, if we introduce this, what might that do? There are teams doing loads and loads of different things, and it is a huge gulf of different things. But we have using just Posit, I think we are edging towards about 600 analysts and they are in a huge variety of areas and also experience levels or kind of focus areas. So there are people who are doing, who are building tooling to use AI models to expose internal documentation better. But there are also people who are just doing kind of routine, almost like an enhanced spreadsheet type work.
It's a huge breadth of things. It's massive, the scope sideways of it. They come when there is a problem shaped like, I wish this machine was bigger.
Do you have a lot of R users who are in RStudio or Positron? Do you have a lot of Python users who are in Positron? What does that sort of composition look like for you as far as the people that you're serving?
It varies a bit between teams. I think we're about 50-50 between or maybe slightly under 50-50. Slightly fewer R users than Python users. And the Python users are mostly between Positron and VS Code, gradually going more Positron. We have got some people using Jupyter still. And we try and support just whatever tools work best for people. So we try as hard as we can not to push people in a particular direction, but to make available the things that are right for their use case and to talk to them about why a certain thing is easy or hard. In terms of things deployed to Connect, the majority these days are Python. I think the future for Connect apps seems like it's quite heavy on Python, at least for the more sort of trailblazing type stuff. But we've got plenty of Shiny and plenty of R as well.
Evaluating and introducing new tools
This question says, when people suggest new tools that they want to use, what is the most helpful information that they share? How does that process go? This says, in short, can you help teach us how to get things approved by our own teams?
The way we tend to work, if people come and say, if I take a simple example, like an extension to Positron, which is pretty easy for us to do, the main questions we'd have is, what's the licensing like? What's the support and how actively maintained is it? And is it solving a unique problem that is not solved by something else that's already available? So something like that is quite simple for us to do. The main reason we don't just blanket allow VS Code extensions is we need to make sure we keep them maintained for the foreseeable future so that we don't pull the rug on anybody.
For more significant kind of technologies, say if we were introducing Connect now as new, the main thing is, what does this give that what you have available now does not provide? Can you build an EC2 server and stick your Python app on it? Why isn't that good enough? What's it bringing that what you have available can't provide now?
And the way we work, the amount of evidence increases with how complex that piece of work is. So an example where we didn't implement it was looking at the inference servers for some of the self-hosted AI models. We were experimenting with those early on before we had access to models provided by cloud providers. And playing with those, we did experiment with whether we would benefit from the inference server. There were some costs, there were some complexities. There is business approval and support type stuff. And that's quite a high bar.
So the effort needs to be worth it. And I think the thing where there's maybe a gap sometimes is when people are asking about new things. They're seeing it in a, I can grab this on my laptop at home. I can make it available. And if I get bored of it in six months, I can get rid of it. Where for quite a few larger organizations in particular, they're looking at it and going, as soon as you put that there, we're running it for 10 years. That is going to be used by someone and we're going to have to maintain it.
As soon as you put that there, we're running it for 10 years. That is going to be used by someone and we're going to have to maintain it.
Dynamic compute and budget allocation
So people have some context around that as well. Because I think it's very cool. So, yeah, that's that's been very popular with people. So that's the thing that we did a small scale rollout of sometime last year. And it's getting gradually getting more adoption now. It's kind of built on it. Well, it's built on top of the Slurm launcher inside Workbench. And what we've done is configured that in such a way that instead of having a static Slurm cluster and slotting things onto it, we launch EC2 instances as people request them. But the thing that's been very popular is in the dropdown, you have a list of instance types and it has the price per hour in dollars. And so an analyst can be given a monthly or project budget. They might choose to use that running a couple of different small instances for a couple of months, or they might choose to run a huge instance for a day. But it puts it empowers them to make those decisions.
And we've built it in such a way that we can sort of follow the org chart. So the senior management can set the overall budget and then team leads can be delegated down to and pass that out as appropriate. And that has been incredibly popular. The initial use case was people who needed to do some particularly large processing on an occasional sort of schedule. And that sort of model is difficult to do with the way large organization budgets are set. But using knowledge of how the organization thinks about budgets, we built a thing where they can go, OK, allocate this much money. That's your pot for the year. Once it's gone, it's gone, which fits the budget cycle and can be then they can use it dynamically as they need.
Training and keeping up with data science
This is an IT perspective, trying to keep track of business as usual, but also trying to make time for data science training. Tom, do you all inside of your sort of admin or IT focus organization train in data science yourselves and try to keep up with what your end users are doing?
We we don't train formally, but as a team, we we have a system of dogfooding. So we have our own tiny Posit Connect tenancy that we use for our own purposes. And what we do is we use the tools that we manage to build internal things like dashboards. So we have things like a release dashboard so people can see changes to the platform over time. We build those things on top of Connect. We use it to do ad hoc analysis type things as we need to. So we sort of bake it into our daily work. And maybe we try and budget in that it's valuable in and of itself to do things like that as part of the team culture. We make sure that we don't overcommit as a team.
For example, instead of analyzing how long it took to do some tickets in Excel where it's familiar, actually taking 10 times longer and doing it in Python or R in order to become familiar with those tools is an acceptable thing to do. There have been a few examples where there would be something that we could do very quickly, where people just use the tools they already know, smash the thing out. But we've managed, we've structured the team in such a way that actually you can implicitly train.
It's difficult to do as an individual contributor. It needs to be an intentional feature of the team and an accepted thing, I think, by everyone involved.
Managing language diversity and knowledge transfer
She says, I noticed that when someone uses programming languages that no one else at the company uses, there can be difficulties when they leave. Either their work needs to be converted to another language or a new person needs to be hired who can use that different languages as it is. How do you minimize this issue? Do you have any sort of thoughts on this from your perspective?
At the organization level, it's it's so big that we sort of don't end up with that problem within the team because we're a relatively small team. So we are there's five of us pretty much. What we would tend to do is we try and minimize it. And I think similar to when introducing a new tool, a similar approach of really justify why you need to bring it on. If it is worth bringing on that new tool, it needs to be more than just a personal preference. So we use a lot of Ansible for things. It's not something I like very much. Not a fan of Ansible, but still using it because it's in wide use across the team and the wider team and is unlikely to go anywhere.
So at that side, try and avoid that problem. If it is worth bringing in, then try and make sure you're sharing work. So something we do within the team is similar to the making the doing it the slow way to learn more things. We try and have people who are less experienced take on areas of the system that they haven't used much before. Again, it's a little bit slower, but we try and avoid having the issue where this person specializes in this area. We still, as you'd expect, we still get things where someone is very familiar. And so we would lean on them if something especially unusual happens. But we will try and spread work around so that people do pick up that language that only one person uses.
And even if it is slower, it tends to be valuable because I think every every language has its own quirks and good sides, bad sides. And I think for anyone who writes code, learning to write another language makes you better at the one you knew before, as well as teaching you about the other one. So if you mostly write R, spending a few weeks being annoyed at Python or vice versa, you'll see new patterns, new approaches, and you will learn something from it. It's not just a time sink.
Upgrading the platform and release cycles
How often do you get to upgrade your Posit environment to bring in new packages, features, etc.? When someone wants to use a package or a new tool, do you have a process that they have to go through to get packages approved and then also versions of R and Python?
So we make those available via package manager and then we have a facility where the individual teams are able to configure allow lists if they want to. So we would make PyPI available with standard scanning stuff like x-ray, filtering out anything that's particularly malicious. But then we have a facility where teams can configure their own allow lists by providing a list of packages and then they can have a filtered view of package manager. And some do that for safety reasons and some do it just so they don't get a sprawl of slightly different implementations.
When it comes to anything that would require an OS package, that we would roll it into our standard release cycle. The way we do releases is we build everything into Docker images just as a convenient deployment artifact. In our development environment, we build several new versions per day and that gets rolled out to dev maybe once to three times a day. Any change like adding a new package just goes through that process and then we apply updated versions to each of our production tenancies on a for the most part monthly basis. But some of our more trailblazing people do every two weeks. We pull in new versions of Posit roughly a week after release for the most part. But for people who want to try things that are a bit newer, we also have an environment that runs the daily builds of Workbench so that when we're chatting to our friends in the kind of account and tech team, if they're telling us about shiny new things, we can just open it and go, I don't like the shade of that.
So, our end users have access to our source code repositories. Some of them have made changes. And, yeah, so, it's not crazy fast, but we can roll a new version out probably within about a month of release to everybody.
So, we tell people via a Teams channel that updates are coming. And we have prearranged maintenance windows for each of the tenancies that align with what their priorities are. There's an agreed it's kind of off the form the third Thursday of the month type schedule. But also, we have kind of particularly engaged people from the Teams. So, the more trailblazer users or the senior kind of senior not quite manager, but the more senior people, they come to our planning sessions and stand-up sessions. So, people are aware of upcoming changes at the point where we're discussing what we're going to do, and they will have feedback at that point.
Often, yeah. So, people will be talking to us about, yeah, things, you know. Often requests or anything other than Posit updates, usually those changes, they have come from users directly logging JIRA tickets, then coming to our planning and discussing where they should be in terms of priority, how broadly applicable those are, how soon they want them, that kind of thing.
Staying close to users
It's not just the formal things like planning. So, we have Teams channels that are for questions and support, and partly those are team specific, and they are for both things like what library would you recommend for this use case, and then people kind of self-answer for what suits their team. We as the platform team sit in those channels and we'll jump, we'll see if people are having challenges with things, and quite often what we might do is just give them a call and watch someone struggle with however it is they're doing things, and then that might be, oh, we could use some help here, or we need to make a change to make it harder for this error to happen.
And actually seeing how people use it, we've learned a lot more about how the system should be, because we're approaching it from system administrator perspective, rather than people who are doing data science things. And, yes, so it's, yes, we might spend two hours on a call with an analyst talking to them about how they are using Python and the issues they're having, and what we make available that might make their lives easier.
There's, I think we're getting close to 600 analysts.
Change management and organizational inertia
No, I wasn't there when it was adopted. I've heard anecdotally that it, formerly the process wasn't, it, our process for adopting new things is, it's slow, but it does kind of rumble through. Getting new things is not that difficult as long as they're solving a completely new problem. I think my understanding is Posit came on during COVID when sort of emergency rules were a bit in effect. But yeah, I think it got in under the, that sort of, we have an urgent problem to solve.
I could say for our routine, for our more routine change. So not something big, like, oh, we'd like to adopt Posit, but for our change process for how we update the platform, it was empathy pretty much. So I come at change management from a traditional I want to do things background. And what are all these people doing who have forms for me? And it took a chunk of time to build empathy and trust between both us and the change management people that us understanding what their priorities are, what they're trying to do, and that actually their remit goes beyond protect the users, make sure there's not an outage. There's a lot more to it than that. There's also the coordinating wider business events where something we don't know about might depend on our system or something we depend on might be changing at the same time, thinking about risks, making sure all of the, you know, impact assessments and all of this stuff is in a sensible place.
And that was quite a long journey. But having gone through that, last time I went into the office, I got a hug from someone who was from the change team having not all of this done. And it's about relationships and empathy, pretty much.
And it's about relationships and empathy, pretty much.
Team performance and capacity
Sometimes it feels like only a few people on a team are stars who really move the needle. As a leader, how do you raise everyone's level so the whole team performs like your top players?
I think something we try and do is we try and do things, particularly when it comes to externally facing, so other stakeholders within the business, we try and act as a team doing these things, and we work in a way that all of us are comfortable with. I don't really like to think of it in terms of people being high or low performers. There's a set of work that needs to happen. People have got some specialisms and preferences, but the team as a unit is trying to get things done. There is a, I feel there's a bit of a risk if you've got too much of the kind of the hero culture type thing that can get you into a bad place. It's a bit of a downer for people who aren't being specifically called out.
I particularly like it when we get praised as a team. Occasionally, we've had feedback from people where they are calling out the team as an entity who are delivering what people need, and I think that's a good thing. You can make sure people are doing the right work that suits them, but I'd also try and think in terms of is the team structured and is it receiving work in the right way that allows everybody the time, the space to succeed, and also the space to be doing things that aren't a strong suit so that they can learn from that. If you're running at 100% capacity all the time, you're in trouble as soon as someone's, you know, they're a bit under the weather or they're worried about something.
If you're at that kind of capacity, it's going to be bad, and maybe to push back a little bit on it, try and have a slower pace of deadlines, or that kind of thing, but that also building trust with the wider organisation really helps with that. So, when we don't get do-it type requests, we get conversations about what's possible, how it might be doable, and we can start by having informal conversations that are a bit that's going to be easy, that's going to be hard. You can have that in a couple of weeks, and this will probably be eight months, and we have these kind of conversations. Sometimes people go, actually, it's not worth that much effort, and you're having honest conversations with stakeholders, and it's not kind of work flowing downhill, it's a collaboration. And some of the collaboration is actually sometimes it's going to take us a bit longer to do things, and it's a lot of trust-building to get to that point.
I think it's trying to avoid the situation is probably the way I would approach that. Well, I don't think I could have asked for a better piece of career advice there, because all of that is such a good idea, but so hard to actually implement, right? Like, stay at 85% capacity, allow yourself that buffer, build coalitions inside of your organisation, build trust, work on things collaboratively. Everything is a good idea until someone has to implement it, and it takes a long time. So, thank you so much for giving us all of your wisdom today.
