The Z-Wave interface I’m working on is an inherently asynchronous beast. Callbacks abound, and the use of lambda functions makes that much easier to deal with. This fact led me to select C++11 as the language standard for the project.

And then I added automatic memory management with std::shared_ptr<>, and it all fell apart.

Why, you ask?

Background: Lambdas in C++

The naïve view of lambdas is that they’re little more than function pointers in a fancy package. Basically, in C++11, you can do something like this and it will work as expected:

So long as those square brackets have nothing between them, this will work fine; the lambda is compatible with a standard function pointer. But if you want to do something more complicated, like capturing variables from the parent scope, things have to look a little different:

This one captures the value of mynum, and will use it when the lambda is called. Note the change in the typedef, however: capturing something changes the datatype of the lambda, and you can no longer assign it to a standard function pointer.

While it may not be immediately obvious, the implications of this are extremely important!

The Problem With Shared Pointers

The Z-Wave protocol is inherently asynchronous, and as a result I use a lot of callbacks in my code. A typical pattern might be something along these lines:

Note that the shared pointer is captured by the callback. I use a similar pattern in a number of places to avoid having to pass pointers around through the callback signature. Unfortunately, this pattern is effectively broken.

It leaks like a sieve.

The object allocated by that std::make_shared<> call? It’ll never get deleted.

Wait, what?

The Truth About Lambdas

Here’s where we come back to that change in type signature when we capture variables with a lambda.

A lambda that captures nothing is simply a function pointer; it doesn’t need to pass any data around. A lambda that captures variables, on the other hand, needs a place to put them. When you create a lambda function that captures variables, you are in essence creating an entirely new kind of object, and the function pointer is just one member of that object; the captured variables are the others.

This means that as long as something holds a reference to your lambda, it also holds indirect references to copies of each element of captured data.

Normally this is not a big deal. Captures are typically copies, so you get a copy of some object and don’t worry about it. Even if you’re not too careful, nothing bad happens outside a bit of extra memory use.

In matters of std::shared_ptr<>, however, it’s a little bit different.

If you capture a shared pointer with a lambda, that lambda will contain a shared pointer, pointing to the same object as the original. It’s just like manually creating two shared pointers. The copy gets passed around with the lambda, ensuring that there’s an extra reference to that object.

If you then store such a lambda on the object for which it has captured a shared pointer, you’ve just created a scenario where the object owns a reference to itself, and thus can never be deleted.

Fun times.

Fixing The Problem

To fix this, first establish a rule: never capture a std::shared_ptr<> in a lambda. It’s an accident waiting to happen, and you’ll be forever trying to chase down that errant reference. When it bites you, it will bite you hard.

There are two safe alternatives you can use.

The most obvious (and the one I do not recommend) is to capture the raw pointer for your lambda to use. This has a major downside, though, in that the object might be deleted before you try to reference it, and you have no way of knowing it. At least it won’t hold an extra reference, though.

The better way (IMO) is to use std::weak_ptr<>.

The std::weak_ptr<> object, by definition, does not hold a reference to the target. It does, however, track the target, so you can tell if the target was deleted. This is perfect for most scenarios where lambdas might be used.

A simple solution would look something like this:

Note that the only change is to the lambda itself. Instead of capturing the shared pointer, it captures a weak reference to the object it points at.

The use of the weak pointer eliminates the potential for self-referencing, and the only cost is that you need to verify the pointer before using it to avoid an accidental null pointer dereference. You don’t need to be concerned about adding an extra reference any longer; even if you store the lambda on the object the shared pointer references, everything will still work perfectly.

It’s an easy solution once you know the problem exists.

And with that, I’m off to do more work on my little toy…



6 comments on “Lambda + shared_ptr<> = memory leak”:
    Johan
    July 5, 2017
    3:27 am

    Here is a little known trick which can make solving this problem a bit easier. You do not need to define and clutter the local scope with an additional std::weak_ptr. Instead you can define it in-place using the following syntax:

    std::shared_ptr obj = std::make_shared();
    obj->on_complete(weak_obj = std::weak_ptr(obj) {
    auto obj = weak_obj.lock();
    if (obj) {
    obj->clean_something_up();
    }
    });

    Sopel
    July 6, 2017
    5:23 am
     

    that’s not valid C++

    Johan
    July 7, 2017
    6:46 am
     
     

    Compiles fine: http://cpp.sh/2pwkz

    Manuel Freiholz
    July 6, 2017
    11:55 pm

    I had this problem a few month ago and also fixed it using the std::weak_ptr approach.

    Nice that you wrote about it so it hopefully helps others 😛

    Thanks.

    Jarek
    July 7, 2017
    2:42 am

    std::weak_ptr is the nicest solution, but there 3rd one.

    Note that the copy of the shared_ptr is stored in my_class::complete_callback field.
    You can “reset” the complete_callback when it is no longer needed:

    obj->on_complete(obj {
    obj->clean_something_up();
    obj->on_complete(my_class::callback());
    });

    Remember to always check if complete_callback is valid before calling it:
    if (complete_callback) complete_callback();

    Sergei
    July 11, 2017
    12:48 am

    You design is broken. You’ve create cycle by yourself by storing calback with shared_ptr. Then you’ve got surprised.

Comments are closed.