Moved this to a draft as I discovered a fairly bad bug in the lambda that will require changes in this script too (if the CodeBuild job spends too much time starting up, the lambda will queue multiple of them).

Ok I fixed the issue I encountered.

The problem was that the lambda waits for the build to start, which might take more than a minute if there are too many queued CodeBuild jobs in the region (I often saw it taking between one and two minutes in the queue).

While the lambda did have a timeout of 15 minutes to account for this, the AWS CLI also has its own socket read timeout of a minute, and by default retries the API invocation three times when it encounters a socket read timeout. So, if the lambda didn't finish within a minute (aka if CodeBuild took more than a minute to provision a builder), AWS CLI would retry starting the lambda, resulting in more than one CodeBuild builder being started.

To address this I explicitly configured the max attempts to be one, and increased the socket read timeout to 15 minutes.

Are we synchronously invoking the lambda? I guess that might be the issue - I'd expect that an async invocation doesn't have a socket timeout tied into the execution progressing?

But the fix sounds good to me.

Are we synchronously invoking the lambda? I guess that might be the issue - I'd expect that an async invocation doesn't have a socket timeout tied into the execution progressing?

Yes we are synchronously invoking it, as we need to retrieve its outputs.